Image processing method and device, and program

ABSTRACT

The present invention relates to a method, apparatus and program for processing an image, capable of producing a sharp image. A digital camera 1 includes an imaging device  4  formed of a single-plane sensor that provides one color signal for each pixel. A plurality of images are captured at a high shutter speed by the imaging device  4  and stored in a frame memory  22  of a signal processor  7.  The signal processor  7  detects a positional relationship among the plurality of images stored in the frame memory  22  and estimates a correct image with three color signals at each pixel, based on the detected positional relationship. The present invention is applicable, for example, to a digital camera.

TECHNICAL FIELD

The present invention relates to a method, apparatus and program forprocessing an image, and more particularly, to a method, apparatus andprogram for processing an image to obtain a sharp image using an imagingdevice including a single-plane sensor.

BACKGROUND ART

In recent years, a digital camera has become very popular as a camera.In the digital camera, an image is taken using an imaging device such asa CCD (Charge Coupled Device) or a CMOS (Complementary Metal OxideSemiconductor) and the image is displayed on a monitor such as a LCD(Liquid Crystal Display) disposed on the digital camera such that a usercan view the image (hereinafter, such an image taken by the imagingdevice will be referred to simply as a captured image). An image signalof the captured image is converted into digital form and transferred toan image processing apparatus such as a personal computer (PC) via astorage medium such as a flash memory or via wired or wirelesscommunication using a cable or an infrared ray. On receiving the imagedata captured by the digital camera, the personal computer displays theimage on a monitor such as a CRT (Cathode Ray Tube) or a LCD (LiquidCrystal Display) so that a user can view the image and edit it.

When an image of a subject that is not well lighted is taken by adigital camera in a darkish environment such as a shaded place or apoorly lighted room, the shutter speed should be set to be low enough toobtain sufficient exposure (that is, the exposure time should be longenough).

When the exposure time is long, if the digital camera is firmly held,for example, by placing it on a tripod such that the digital camera doesnot shake, a good image can be obtained. However, in a case in which thedigital camera is held in hands, the digital camera can shake. If thecamera shakes when the shutter is in an open state (during a shuttertime), a resultant captured image of a subject is blurred. The blurredimage is called a “hand shake blurred” image or a “camera shake blurred”image.

It is known to obtain equivalently sufficient exposure by simplycumulatively superimposing two or more captured images, instead ofdecreasing the shutter speed (increasing the shutter time) (an exampleof such a technique may be found, for example, in Japanese UnexaminedPatent Application Publication No. 05-236422).

However, in the technique disclosed in Japanese Unexamined PatentApplication Publication No. 05-236422, because two or more capturedimages are simply superimposed, an image is blurred if the digitalcamera shakes, as in the case in which an image is captured at a lowshutter speed.

In some digital cameras available from Canon Kabushiki Kaisha, atechnique called an image stabilizer (IS) is used to prevent an imagefrom being blurred when a digital camera shakes.

In the technique using the image stabilizer, a shake or vibration of adigital camera is detected by a pre-sensor disposed in an optical lenssystem, and a part (correction optical system) of the lens system ismoved in a direction perpendicular to the optical axis in accordancewith a digital signal indicating the detected shape or vibration of thedigital camera so that refraction of light is changed to prevent theimage from being blurred.

The image stabilizer makes it possible to cancel out blurring of animage due to shaking of a camera or a stage on which the camera isplaced caused by shaking of a hand or a wind, which is significant inparticular when an image is taken at a low shutter speed by a digitalcamera held in hands, thereby allowing a user to obtain a sharp image.

However, the image stabilizer needs a dedicated sensor responsible forsensing shaking of the camera and also needs a mechanism for quicklymoving a part (correction optical system) of the lens system, and thusthe image stabilizer has a complicated structure, which results in anincrease in production cost.

Another known technique to prevent an image from being blurred when theimage is captured by a shaking camera is to successively capture aplurality of images at a high shutter speed, detect the displacement ofsecond and following captured images relative to a first captured image,compensate for the displacement of the second and following capturedimages, and superimposing the compensated images on the first capturedimage (examples of this technique may be found, for example, in JapaneseUnexamined Patent Application Publications Nos. 2000-217032,2000-224460, 2000-244803, 2000-244797, 2000-069352, 10-341367, and09-261526). Basically, in this conventional technique, images similar toa first captured image are produced from second and following capturedimages by means of correction or interpolation, and the resultant imagesare simply added or linearly added on a pixel-by-pixel basis to thefirst captured image.

In this technique, each of images successively captured at a highshutter speed has practically no blur although images are dark. An imagefinally obtained by superimposing of second and following capturedimages on a first image can have high brightness similar to thatobtained when the exposure time is set properly.

In this technique, the correction of the detected displacement of thesecond and following captured images relative to the first capturedimage is made by interpolating color signals (data) including a R (Red)signal (red data), a G (Green) signal (green data), and a B (Blue)signal (blue data) for each pixel, by using an interpolation functionbased on linear interpolation, bicubit interpolation, or the like.

However, in the case of an imaging device using a single-plane sensor asan imaging device, each pixel outputs only one of color signals, thatis, an R signal, a G signal, or a B signal. This makes it impossible tocorrect displacements of a plurality of images by means of interpolationdescribed above, and any interpolation method applicable to thesingle-plane sensor is known.

DISCLOSURE OF INVENTION

In view of the above, it is an object of the present invention toprovide a technique of obtaining a sharp image captured by, for example,an imaging device using a single-plane sensor even in a state in whichcamera shake occurs.

The present invention provides an image processing method, comprisingthe steps of detecting a positional relationship among a plurality ofinput images captured by image capture means for capturing an image eachpixel of which has one of n pixel values, and estimating an output imageby identifying observed pixel components that should be taken intoconsideration from the plurality of input images for each pixel positionof the output image, and producing the output image having (n+1) or morepixel values for each pixel based on the identified observed pixelcomponents.

In the step of estimating the output image, observed pixel componentsthat should be taken into consideration may be selected from theplurality of input images for each pixel position of the output image,and the output image having (n+1) or more pixel values for each pixelmay be produced based on the selected observed pixel components.

In the step of estimating the output image, the output image having(n+1) or more color signals for each pixel may be produced from theplurality of input images, based on the positional relationship detectedin the detection step.

In the step of estimating the output image, the output image may bedetermined taking into account a color correlation at each pixelposition and/or in the vicinity of each pixel position of the outputimage. In the step of estimating the output image, pixel values ofpixels of the input images may be changed to pixel values depending onthe degree of steepness of an edge, and the output image may beestimated based on the resultant changed pixel values. n may be equal to1, and in the step of estimating the output image, the output imagehaving three pixel values for each pixel may be produced from theplurality of input images, based on the positional relationship detectedin the detection step.

The image capture means may be a single-plane sensor, in which a colorsignal observed varies depending on the position of each input image.

The image capture means may capture the plurality of input images withunderexposure, and the image processing method may further comprise thestep of correcting the exposure by gaining up each pixel value of theplurality of input images.

The step of estimating the output image may comprise the steps ofdetermining a corrected position of each pixel depending on thepositional relationship detected in the detection step for each pixel ofeach of the plurality of input images, producing a light intensityintegral constraint formula indicating that the difference between theintegral of light intensity in a region centered at the correctedposition and a pixel value of a pixel at the corrected position shouldbe less than a predetermined error; and calculating the light intensityintegral constraint formula.

When the light intensity integral constraint formula is produced, if theplurality of input images have been gained up, the error may be changeddepending on the factor of the gaining up.

In the step of estimating the output image, a corrected position of eachpixel may be determined depending on the positional relationshipdetected in the detection step for each pixel of each of the pluralityof input images, and the output image may be determined taking intoaccount a pixel value of a pixel of the input images whose correctedposition is located in the vicinity of each pixel position of the outputimage and also taking into account the error depending on the factor ofthe gaining up.

In the step of estimating the output image, a corrected position of eachpixel may be determined depending on the positional relationshipdetected in the detection step for each pixel of each of the pluralityof input images, and the output image may be determined taking intoaccount a pixel value of a pixel of the input images whose correctedposition is located in the vicinity of each pixel position of the outputimage and also taking into account the distance between the correctedposition and the position of the corresponding pixel of the outputimage.

In the step of estimating the output image, a corrected position of eachpixel may be determined depending on the positional relationshipdetected in the detection step for each pixel of each of the pluralityof input images, and the output image may be determined taking intoaccount a pixel value of a pixel of the input images whose correctedposition is located in the vicinity of each pixel of the output imageand also taking into account the error depending on the factor of thegaining up.

In the step of estimating the output image, a corrected position of eachpixel may be determined depending on the positional relationshipdetected in the detection step for each pixel of each of the pluralityof input images, and the output image may be estimated by determiningthe weighted sum of pixel values of the input images whose correctedposition is located in the vicinity of each pixel position of the outputimage.

In the step of estimating the output image, the correct pixel value of apixel of interest may be determined by determining the weighted sum ofpixel values of the input images whose corrected position is located inthe vicinity of each pixel position of the output image, weights of theweighted sum being determined using a function having a lowpass-filtercharacteristic with respect to the distance between the position of thepixel of interest in the output image and the corrected position.

In the step of estimating the output image, the correct pixel value ofthe pixel of interest may be estimated by selectively performing a firstcalculation process of determining the weighted sum of pixel values ofthe input images whose corrected position is located in the vicinity ofthe position of the pixel of interest, or a second calculation processof determining the weighted sum of pixel values of the input imageswhose corrected position is located in the vicinity of the position of apixel of the output image located adjacent to the pixel of interest.

In the step of estimating the output image, the second calculationprocess may be selected when the sum of weights in the first calculationprocess is equal or nearly equal to 0.

In the step of estimating the output image, the correct pixel value ofthe pixel of interest may be determined by selectively performing afirst calculation process of determining the weighted sum of pixelvalues of the input images whose corrected position is located in thevicinity of the position of the pixel of interest, or a secondcalculation process of determining the weighted sum of pixel values ofthe input images whose corrected position is located in a secondvicinity of the position of the pixel of interest, the second vicinitybeing greater than the first vicinity.

In the step of estimating the output image, the second calculationprocess may be selected when the sum of weights in the first calculationprocess is equal or nearly equal to 0.

In the step of estimating the output image, the correct pixel value ofthe pixel of interest may be determined by selectively performing afirst calculation process of determining the sum of pixel valuesweighted by a first weighting factor of the input images whose correctedposition is located in the vicinity of the position of the pixel ofinterest, or a second calculation process of determining the sum ofpixel values weighted by a second weighting factor of the input imageswhose corrected position is located in the vicinity of the position ofthe pixel of interest.

In the step of estimating the output image, the second calculationprocess may be selected when the sum of weights in the first calculationprocess is equal or nearly equal to 0.

In the detection step, the positional relationship among the pluralityof input images may be detected with reference to an input imagecaptured at the exact or nearly exact center of a time during which theplurality of input images were captured.

The present invention provides an image processing apparatus comprisingimage capture means for capturing a plurality of input images having npixel values for each pixel; detection means for detecting a positionalrelationship among the plurality of input images, and means forestimating an output image by identifying observed pixel components thatshould be taken into consideration from the plurality of input imagesfor each pixel position of the output image, and producing the outputimage having (n+1) or more pixel values for each pixel based on theidentified observed pixel components.

The present invention provides a program for causing a computer toexecute a process comprising the steps of detecting a positionalrelationship among a plurality of input images captured by image capturemeans for capturing an image each pixel of which has one of n pixelvalues, and estimating an output image by identifying observed pixelcomponents that should be taken into consideration from the plurality ofinput images for each pixel position of the output image, and producingthe output image having (n+1) or more pixel values for each pixel basedon the identified observed pixel components.

According to the present invention, a positional relationship among aplurality of input images having n pixel values at each pixel isdetected. Of observed pixel components of the plurality of input images,observed pixel components that should be taken into consideration areidentified for each pixel position of an output image, based on thedetected positional relationship. Based on the observed pixel componentsidentified, an output image having (n+1) or more pixel values at eachpixel is.estimated.

The image processing apparatus may be in the form of a self-containedapparatus or a block responsible for image processing in an apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing anexample of a configuration of a digital camera 1 according to anembodiment of the present invention.

FIG. 2 is a flow chart of a process performed by the digital camera 1shown in FIG. 1 to take an image.

FIG. 3 is a diagram showing a manner in which pixels of an imagingdevice 4 shown in FIG. 1 are arranged.

FIG. 4 is a block diagram showing an example of detailed configurationof a signal processor 7 shown in FIG. 1.

FIG. 5 is a diagram showing a relationship in terms of coordinatesbetween a first image and a second image.

FIG. 6 a flow chart showing an image estimation process according to afirst embodiment.

FIG. 7 a flow chart showing an image estimation process according to thefirst embodiment.

FIG. 8 is a diagram showing a color correlation constraint according toa second embodiment.

FIG. 9 is a diagram showing a color correlation constraint according toa second embodiment.

FIG. 10 a flow chart showing an image estimation process according to asecond embodiment.

FIG. 11 a flow chart showing an image estimation process according tothe second embodiment.

FIG. 12 a flow chart showing an image estimation process according tothe second embodiment.

FIG. 13 is a diagram showing a first image stored in a frame memory22-1.

FIG. 14 is a diagram showing a second image stored in a frame memory22-2.

FIG. 15 is a diagram showing a third image stored in a frame memory22-3.

FIG. 16 is a diagram showing pixels of the imaging device 4.

FIG. 17 is a diagram showing a manner in which pixels of the imagingdevice 4 shown in FIG. 1 are arranged.

FIG. 18 is a diagram showing a spring model.

FIG. 19 is a diagram showing a spring model.

FIG. 20 is a diagram showing a spring model.

FIG. 21 is a diagram showing a spring model.

FIG. 22 is a diagram showing a spring model.

FIG. 23 is a diagram showing a spring model.

FIG. 24 a flow chart showing an image estimation process according to athird embodiment.

FIG. 25 a flow chart showing an image estimation process according to afourth embodiment.

FIG. 26 a flow chart showing a correct image estimation processaccording to a fifth embodiment.

FIG. 27 is a diagram showing a state in which green components (Gsignals) are sensed by respective pixels of the imaging device 4.

FIG. 28 is a diagram showing an edge extending in a vertical direction.

FIG. 29 is a diagram showing an edge extending in a horizontaldirection.

FIG. 30 is a diagram showing an edge extending in a direction from upperleft to lower right.

FIG. 31 is a diagram showing an edge extending in a direction from upperright to lower left.

FIG. 32 is a diagram showing a plane Q1 produced when an edge extendingin the vertical direction is detected.

FIG. 33 is a diagram showing a plane Q2 produced when an edge extendingin the horizontal direction is detected.

FIG. 34 is a diagram showing a plane Q3 produced when an edge extendingin the direction from upper left to lower right.

FIG. 35 is a diagram showing a plane Q4 produced when an edge extendingin the direction from upper right to lower left.

FIG. 36 a flow chart showing an image estimation process according to asixth embodiment.

FIG. 37 is a diagram showing a color correlation constraint according toa seventh embodiment.

FIG. 38 is a diagram showing a color correlation constraint according tothe seventh embodiment.

FIG. 39 a flow chart showing an image estimation process according tothe seventh embodiment.

FIG. 40 is a diagram showing the difference between the presentinvention and a conventional technique.

FIG. 41 is a block diagram showing an example of a configuration of acomputer according to an embodiment of the present invention.

FIG. 42 is a diagram showing a cubic function.

FIG. 43 is a diagram showing pixel values at a position (I′, J′) in areference coordinate system.

FIG. 44 is a diagram showing pixel values in an exceptional state at aposition (I′, J′) in the reference coordinate system.

FIG. 45 is a diagram showing an exception handling process for a Gsignal.

FIG. 46 is a diagram showing an exception handling process for a Gsignal.

FIG. 47 is a diagram showing an exception handling process for a Rsignal.

FIG. 48 is a diagram showing an exception handling process for a Rsignal.

FIG. 49 a flow chart showing an image estimation process according to aneighth embodiment.

FIG. 50 a flow chart showing an image estimation process according tothe eighth embodiment.

FIG. 51 a flow chart showing an image estimation process according tothe eighth embodiment.

FIG. 52 a flow chart showing an image estimation process according tothe eighth embodiment.

FIG. 53 a flow chart showing an image estimation process according tothe eighth embodiment.

FIG. 54 is a diagram showing captured images 401 ₁ to 40 ₁₈.

FIG. 55 is a diagram showing an output image produced from capturedimages by using a first captured image as a reference image.

FIG. 56 is a diagram showing an output image produced from capturedimages by using an intermediate captured image as a reference image.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 is a block diagram showing an example of a configuration of adigital (still) camera 1 according to an embodiment of the presentinvention.

The digital camera 1 shown in FIG. 1 includes a lens 2, an aperture 3,an imaging device 4, a correlated double sampler 5, a A/D(Analog/Digital) computer 6, a signal processor 7, a timing generator 8,a D/A (Digital/Analog) converter 9, a video encoder 10, a monitor 11, acodec 12, a memory 13, a bus 14, a CPU (Central Processing Unit) 15, andan input device 16. The A/D converter 6 includes a shifter 12, and thesignal processor 7 includes a frame memory 22.

Light originating from a subject (not shown) passes through an opticalsystem including the lens 2 and the aperture 3, and is incident on theimaging device 4. The imaging device 4 is in the form of a single-planesensor made of a CCD or a CMOS including a particular number of pixels(picture elements).

The imaging device 4 senses the incident light originating from thesubject for a specified period (shutter time) at specified intervals inaccordance with an exposure timing signal supplied from the timinggenerator 8. The imaging device 4 converts the intensity of light sensedby each of photosensor elements disposed in an image sensing plane to anelectric signal by means of photoelectric conversion. The resultantelectric signal is supplied as an image signal to the correlated doublesampler 5. Because the imaging device 4 is constructed in the form ofthe single-plane sensor, each pixel of the image supplied in the form ofthe electric image signal to the correlated double sampler 5 is one ofR, G, and B signals (data).

The imaging device 4 captures a plurality of images (N images) at ahigher shutter speed (with a longer shutter time) than a shutter speed(shutter timer (exposure time)) necessary to obtain optimum exposure forone image, for use to produce a sharp output image even if a camerashake occurs. Each of N images (input images) captured (withunderexposure) by the imaging device 4 is darker than is obtained if theimage is captured with optimum exposure. That is, the brightness of eachimage is lower by a factor of Mk (k=1 to N) than that obtained withoptimum exposure (that is, equal to 1/Mk times that obtained withoptimum exposure). The value of Mk is determined by the shutter speed.

The correlated double sampler 5 removes noise from the image signal(electric signal) received from the imaging device 4 by means ofcorrelated double sampling and supplies the resultant image signal tothe A/D converter 6. On receiving the noise-removed image signal of thesubject from the correlated double sampler 5, the A/D converter 6converts it into a digital signal, that is, the A/D converter 6 samplesthe received image signal and quantizes it. Thereafter, the shifter 21increases (gains up) the magnitude of the digital image signalrepresenting a dark image captured with underexposure by a factor of Mk,for example, by means of n′-bit shifting so as to obtain an image withbrightness (signal level) similar to that obtained with optimumexposure. The resultant image signal is supplied to the signal processor7.

Note that the correlated double sampler 5 cannot remove all noisecomponents in the above-described process of removing noise from theimage signal, and thus the image signal output from the correlateddouble sampler 5 includes a residual noise component. The residual noisecomponent remaining without being removed by the correlated doublesampler 5 causes an error for the true value of the image signal, andthe error is also increased by the shifter 21 by a factor of Mk as wellas the image signal. Thus, the resultant error depends on the factor ofthe gaining up performed by the shifter 21. Let E denote the magnitudeof the residual noise component including the image signal output fromthe correlated double sampler 5. The noise E is set to a maximumpossible value depending on the characteristic of the imaging device 4.The image signal supplied from the A/D converter 6 to the signalprocessor 7 includes noise with a magnitude approximately equal to Mktimes the noise E (E×Mk). For example, when Mk=8, n′=set to be 3, andthe shifter 21 performs 3-bit shifting on the k-th captured image toincrease the brightness of the k-th captured image to proper brightness.

After the image signal is gained up by the factor of Mk such that thebrightness is increased to the optimum level, the image signal of eachof N captured images is output from the A/D converter 6 and temporarilystored in the frame memory 22 of the signal processor 7.

In accordance with programs prepared in advance, the signal processor 7performs a process on the image signal of each of N captured images withbrightness gained up to the proper level stored in the frame memory 22.

More specifically, the signal processor 7 employs the first one of the Ncaptured image as a reference image, and sequentially selects second andN-th captured image as a target image. The signal processor 7 detectsthe positional displacement of the target image relative to thereference image. The signal processor 7 compensates for the camera shakebased on the detected positional displacement and produces adisplacement-compensated sharp image (to be output) having all G, R, andB signals for each pixel. The resultant output image signal is suppliedto one or both of the D/A converter 9 and codec 12. The signal processor7 may be realized using a DSP (Digital Signal Processor) or the like. Inthe following discussion, it is assumed that the image signal processedby parts following the A/D converter 6 has been gained up in brightnessto a level similar to that obtained with optimum exposure.

The timing generator 8 supplies the exposure timing signal to theimaging device 4, the correlated double sampler 5, the A/D converter 6,and the signal processor 7 such that N images are captured at thespecified intervals. An use may change the interval, for example,depending on the brightness of a subject. If the interval is changed bythe user via the input device 16, the data indicating the changed valueof the interval is supplied from the CPU 15 to the timing generator 8via the bus 14.

The D/A converter 9 converts the image signal output from the signalprocessor 7 into an analog signal and supplies the resultant analogimage signal to the video encoder 10. The video encoder 10 converts theimage signal (in the form of the analog signal) supplied from the D/Aconverter 9 into a video signal in a form capable of being displayed onthe monitor 11. The resultant video signal is supplied to the monitor11. The monitor 11 serves as a viewfinder of the digital camera 1, andit may be realized using a LCD or a CRT. The video signal supplied fromthe video encoder 10 is displayed on the monitor 11. Thus, a sharp imageis displayed on the monitor 11.

The codec 12 encodes the image signal output from the signal processor 7in according to a particular coding technique such as that according tothe JPEG (Joint Photographic Exerts Group) standard, the MPEG (MovingPicture Experts Group) standard, or the DV (Digital Video) standard. Theresultant encoded image signal is supplied to the memory 13.

The memory 13 is made of a semiconductor memory such as a flash memoryand servers to temporarily or permanently store the encoded image signalsupplied from the codec 12. Instead of the memory 13, a storage mediumsuch as a magnetic disk or an optical (magnetooptical) disk may be used.The memory 13 or the storage medium used instead of the memory 13 may beremovable from the digital camera 1. The digital camera 1 may includeboth a storage medium disposed in a fixed fashion in the digital camera1 and a storage medium disposed in a removable fashion on the digitalcamera 1.

The CPU 15 supplies a control signal to various parts via the bus 14 tocontrol various processes. For example, in accordance with an imagecapture start signal issued by a user via the input device 16, the CPU15 supplies the control signal to various parts to capture an image of asubject and store the captured image in the memory 13.

The input device 16 includes operation buttons such as a release buttondisposed on the main part of the digital camera 1. If an operationbutton is operated by a user, a signal is generated and supplied to theCPU 15 via the bus 14. In accordance with the signal received from theinput device 16 via the bus 14, the CPU 15 controls various parts. Oneor more of the operation buttons of the input device 16 may be displayedon the monitor 11. The operation imposed on an operation buttondisplayed on the monitor 11 may be detected by a transparent tabletdisposed on the monitor 11.

Referring to a flow chart shown in FIG. 2, a process of taking an imageperformed by the digital camera 1 is described below.

First, in step S1, the imaging device 4 takes an image of a subject.More specifically, each time the release button (shutter button) ispressed, the imaging device 4 successively captures N images at a highshutter speed by sensing incident light originating from the subject inaccordance with the exposure timing signal supplied from the timinggenerator 8. Thus, each time an image is taken, dark N captured imageswith a brightness level lower than that obtained with optimum exposureare obtained. The light originating from the subject is sensed andconverted into an electric signal. The resultant image signal in theform of the electric signal is supplied to the correlated double sampler5. The correlated double sampler 5 removes a noise component from theimage signal and supplies the resultant image signal to the A/Dconverter 6. Thereafter, the process proceeds to step S2.

In step S2, the D/A converter 6 converts the noise-removed image signalof the subject received from the correlated double sampler 5 into adigital signal. Furthermore, the shifter 21 converts the underexposeddark image into an image with brightness (signal level) similar to thatobtained with optimum exposure by means of n′-bit shifting. Theresultant image signal is supplied to the signal processor 7.Thereafter, the process proceeds to step S3.

In step S3, the signal processor 7 takes the first captured image as areference image and sequentially selects one of image following thefirst image as a target image. The signal processor 7 detects apositional displacement (amount of motion) of the target image (one ofsecond to N-th images) with respect to the reference image. Thereafter,the process proceeds to step S4.

In step S4, the signal processor 7 performs the correct image estimationprocess, based on the positional displacement of the N-th captured imagerelative to the reference image detected in step S3. Thereafter, theprocess proceeds to step S5. In this correct image estimation processthe details of which will be described later, the signal processor 7produces a displacement-compensated sharp image (output image) includingall G, R, and B signals for each pixel and supplies the resultant imagesignal to one or both of the D/A converter 9 and codec 12.

In step S5, the output image is displayed on the monitor 11, and theoutput image is stored in the memory 13 such as a flash memory. Thus,the process is completed. More specifically, in step S5, the imagesignal supplied in step S4 from the signal processor 7 to the D/Aconverter 9 is converted into an analog signal and supplied to the videoconverter 10. Furthermore, in step S5, the video encoder 10 converts theanalog image signal supplied from the D/A converter 9 into a videosignal in a form capable of being displayed on the monitor 11. Theresultant video signal is supplied to the monitor 11. Furthermore, instep S5, the monitor 11 displays the image in accordance with the videosignal supplied from the video encoder 10. Furthermore, in step S5, theimage signal supplied in step S4 from the signal processor 7 to thecodec 12 is encoded according to an encoding scheme such as that basedon the JPEG standard or the MPEG standard, and the resultant imagesignal is stored in the memory 13 such as a flash memory. Thus, theprocess is exited.

FIG. 3 is a diagram showing a manner in which pixels of the imagingdevice 4 are arranged. Although only some pixels (6 pixels in ahorizontal direction by 4 pixels in a vertical direction, and thus atotal of 24 pixels) in an upper left area of the imaging device 4 areshown in FIG. 3, the remaining pixels are arranged in a similar manner.

In FIG. 3, an X-Y coordinate system is defined such that the upper leftcorner of the imaging device 4 is employed as an origin, an X axis istaken in a horizontal direction (to the right), and a Y axis is taken ina vertical direction (upwardly). The horizontal length and the verticallength of one pixel are set to be 1. The position (center position) ofthe pixel at an i-th position from the left end and at a j-th positionfrom the upper end can be expressed as (i−0.5, j−0.5). In the imagingdevice 4 shown in FIG. 3, pixels are arranged in the form of a Bayerarray.

Pixels that provide G signals include a pixel G00 at a first position inthe X direction and at a first position in the Y direction, a pixel G02at a third position in the X direction and at the first position in theY direction, a pixel G04 at a fifth position in the X direction and atthe first position in the Y direction, a pixel G11 at a second positionin the X direction and at a second position in the Y direction, andfurther a pixel G13, a pixel G15, a pixel G20, a pixel G22, a pixel G24,a pixel G31, a pixel G33, and a pixel G35.

Pixels that provide R signals include a pixel R01 at the second positionin the X direction and at the first position in the Y direction, a pixelR03 at a fourth position in the X direction and at the first position inthe Y direction, a pixel R05 at a sixth position in the X direction andat the first position in the Y direction, a pixel R21 at the secondposition in the X direction and at a third first position in the Ydirection, and further a pixel R23, and a pixel R25.

Pixels that provide B signals include a pixel B10 at the first positionin the X direction and at the second position in the Y direction, apixel B12 at the third position in the X direction and at the secondposition in the Y direction, a pixel B14 at the fifth position in the Xdirection and at the second position in the Y direction, a pixel B30 atthe first position in the X direction and at the fourth position in theY direction, and further a pixel B32, and a pixel B34.

For a sharp image with no blur due to a camera shake, a G signal, a Rsignal, and a B signal at a particular position (x, y) in the X-Ycoordinate system of the imaging device 4 are respectively denoted byLg(x, y), Lr(x, y), and Lb(x, y). Furthermore, for a sharp image with noblur due to a camera shake, a G signal, a R signal, and a B signal of a(i-th, j-th) pixel located at an i-th position as counted from the leftend and at a j-th position as counted from the upper end arerespectively denoted by Lg(i, j), Lr(i, j), and Lb(i, j). That is, Lg(x,y), Lr(x, y), and Lb(x, y) (Lg(i, j), Lr(i, j), and Lb(i, j))respectively indicate the true green light intensity (data), the truered light intensity (data), the true blue light intensity (data) at theparticular position (x, y) (at the (i-th, j-th) pixel) of an imagehaving neither blur nor noise. Hereinafter, Lg(x, y), Lr(x, y), andLb(x, y) (Lg(i, j), Lr(i, j), and Lb(i, j)) will be referred to as thetrue green light intensity Lg(x, y) (Lg(i, j)), the true red lightintensity Lr(x, y) (Lr(i, j)), and the true blue light intensity Lb(x,y) (Lb(i, j)). When x=i−0.5 and y=j−0.5, Lg(x, y)=Lg(i, j), Lr(x,y)=Lr(i, j), and Lb(x, y)=Lb(i, j).

Although in the present embodiment, the pixels of the imaging device 4are arranged in the form of a Bayer array, the pixels may be arranged inanother fashion.

Variables ig, jg, ir, jr, ib, and jb associated with pixels of the Gsignal, the R signal, and the B signal are defined as follows.

Variables ig and jg respectively indicate a position i in the Xdirection and a position j in the Y direction at which a pixelresponsible for providing a G signal is located. That is, combinationsof variables ig and jg are equal to combinations of variables i and jfor pixels responsible for providing the G signal. In the case in whichpixel are arranged in the form of a Bayer array, variables ig and jg aregiven by variables i and j whose difference i−j is even. Note that, as amatter of course, the difference between variables ig and jg, that is,ig−jg, is also even. Thus, a “pixel located at an ig-th position and ata jg-th position” is a pixel that is responsible for providing the Gsignal. In a case in which pixels are arranged in the form of an arrayother than the Bayer array, variables ig and jg are given differentlydepending on the array.

Variables ir and jr respectively indicate a position i in the Xdirection and a position j in the Y direction at which a pixelresponsible for providing a R signal is located. That is, combinationsof variables ir and jr are equal to combinations of variables i and jfor pixels responsible for providing the R signal. In the case in whichpixel are arranged in the form of the Bayer array, variables ir and jrare given by variables i and j that satisfy the constraint that thevariable i is even and the difference between variables i and j, thatis, i−j is odd. Note that, as a matter of course, the difference betweenvariables ir and jr, that is, ir−jr, is also odd. Thus, a “pixel locatedat an ir-th position and at a jr-th position” is a pixel that isresponsible for providing the R signal. In a case in which pixels arearranged in the form of an array other than the Bayer array, variablesir and jr are given differently depending on the array.

Variables ib and jb respectively indicate a position i in the Xdirection and a position j in the Y direction at which a pixelresponsible for providing a B signal is located. That is, combinationsof variables ib and jb are equal to combinations of variables i and jfor pixels responsible for providing the B signal. In the case in whichpixel are arranged in the form of the Bayer array, variables ib and jbare given by variables i and j that satisfy the constraint that thevariable i is odd and the difference between variables i and j, that is,i−j is odd. Note that, as a matter of course, the difference betweenvariables ib and jb, that is, ib−jb, is also odd. Thus, a “pixel locatedat an ib-th position and at a jb-th position” is a pixel that isresponsible for providing the B signal. In a case in which pixels arearranged in the form of an array other than the Bayer array, variablesib and jb are given differently depending on the array.

Values (pixel values) of color signals (a G signal, a R signal, and a Bsignal) output from respective pixels of the imaging device 4 shown inFIG. 3 are defined as follows.

As described earlier, N images are taken by the imaging device 4, andthus N pixel values are obtained for each pixel of the imaging device 4.A pixel value of a pixel at an ig-th position and at a jg-th position ofa k-th image (k=1 to N) is denoted by Gobs(k, ig, jg), a pixel value ofa pixel at an ir-th position and at a jr-th position of a k-th image(k=1 to N) is denoted by Robs(k, ir, jr), and a pixel value of a pixelat an ib-th position and at a jb-th position of a k-th image (k=1 to N)is denoted by Bobs(k, ib, jb). For example, a pixel value of a pixel G00of a first image is given by Gobs(1, 1, 1), and a pixel value of a pixelG04 of a second image is given by Gobs(2, 5, 1). Herein, and also in thefollowing description, k denotes an integer in the range from 1 to N,unless otherwise specified.

In other words, pixel values Gobs(k, ig, jg), Robs(k, ir, jr), Bobs(k,ib, jb) respectively indicate pixel values obtained by pixelsG(jg−1)(ig−1), R(jr−1)(ir−1), and B(jb−1)(ib−1).

FIG. 4 shows an example of detailed configuration of a part of thesignal processor 7 shown in FIG. 1. The signal processor 7 includes aframe memory 22, a motion detector 23, a calculation circuit 24, and acontroller 25. The frame memory 22 includes frame memories 22-1 to 22-N,and the motion detector 23 includes motion detectors 23-1 to 23-(N−1).

As described earlier, the A/D converter 6 supplies N captured images tothe frame memory 22. The frame memory 22-1 stores a first captured imagesupplied from the A/D converter 6. The frame memory 22-2 stores a secondcaptured image supplied from the A/D converter 6. Similarly, the framememory 22-k stores a k-th captured image supplied from the A/D converter6 (where k=3 to N).

The frame memory 22-1 supplies the first captured image stored thereinto the calculation circuit 24 and the motion detectors 23-1 to 23-(N−1)with proper timing. The frame memory 22-2 supplies the second capturedimage stored therein to the calculation circuit 24 and the motiondetector 23-1 with proper timing. Similarly, the frame memory 22-ksupplies the k-th captured image stored therein to the calculationcircuit 24 and the motion detector 23-(k−1) with proper timing.

The motion detector 23 detects a positional relationship between twocaptured images. More specifically, the motion detector 23 takes thefirst captured image as a reference image and one of captured imagesfollowing the first captured image as a target image, and the motiondetector 23 detects a positional displacement (amount of motion) of thetarget image (one of second to N-th images) with respect to thereference image. For example, a camera shake can cause a positionaldisplacement.

The first captured image used as the reference image is supplied to themotion detector 23-1 from the frame memory 22-1, and the second capturedimage used as the target image is supplied to the motion detector 23-1from the frame memory 22-2.

The motion detector 23-1 detects the correspondence between the positionof each pixel (or each block when the image is divided into a pluralityof blocks) of the second captured image and the position of the firstcaptured image by determining conversion parameters (θ, T2x, T2y, S2) ofequation (1) representing the positional relationship between the firstcaptured image and the second captured image, where θ2 denotes arotation angle, S2 denotes a scale, and (T2x, T2y) denotes a paralleldisplacement. The resultant conversion parameters are supplied to thecalculation circuit 24. $\begin{matrix}{\begin{pmatrix}{X1}_{(2)} \\{Y1}_{(2)}\end{pmatrix} = {{{{S2}\begin{pmatrix}{\cos({\theta 2})} & {- {\sin({\theta 2})}} \\{\sin({\theta 2})} & {\cos({\theta 2})}\end{pmatrix}}\begin{pmatrix}{X2} \\{Y2}\end{pmatrix}} + \begin{pmatrix}{T2x} \\{T2y}\end{pmatrix}}} & (1)\end{matrix}$

Equation (1) is in the form of a so-called affine transformation, inwhich (X2, Y2) represents the position of a pixel of the second capturedimage, and (X1 ₍₂₎, Y1 ₍₂₎) represents the corresponding position on thefirst captured image. Note that a subscript (2) is used to indicate aconverted position of a position on the second captured image.

The conversion parameters (θ2, T2x, T2y, S2) represent that a position(X2, Y2) on the second captured image is displaced with respect to acorresponding position (X1, Y1) on the first captured image by an angleθ2 due to a camera shake, enlarged (or reduced if S2<1) by a factor ofS2 due to movement of the digital camera 1 toward a subject, anddisplaced parallel to the subject by (T2x, T2y). In most cases, ofvarious displacement components of camera shake, a component in adirection parallel to the plane of the imaging device 4 has asignificant effect, but a displacement component in a direction from thedigital camera 1 to the subject is small enough to neglect it. That is,S2 can be approximated as S2=1.

The first captured image used as the reference image is supplied to themotion detector 23-2 from the frame memory 22-1, and the third capturedimage used as the target image is supplied to the motion detector 23-2from the frame memory 22-3.

The motion detector 23-2 detects the correspondence between the positionof each pixel (or each block when the image is divided into a pluralityof blocks) of the third captured image and the position of the firstcaptured image by determining conversion parameters (θ3, T3x, T3y, S3)of equation (1) representing the positional relationship between thefirst captured image and the second captured image, where θ3 denotes arotation angle, S3 denotes a scale, and (T3x, T3y) denotes a paralleldisplacement. The resultant conversion parameters are supplied to thecalculation circuit 24. $\begin{matrix}{\begin{pmatrix}{X1}_{(3)} \\{Y1}_{(3)}\end{pmatrix} = {{{{S3}\begin{pmatrix}{\cos({\theta 3})} & {- {\sin({\theta 3})}} \\{\sin({\theta 3})} & {\cos({\theta 3})}\end{pmatrix}}\begin{pmatrix}{X3} \\{Y3}\end{pmatrix}} + \begin{pmatrix}{T3x} \\{T3y}\end{pmatrix}}} & (2)\end{matrix}$

Equation (2) is also in the form of an affine transformation, in which(X3, Y3) represents the position of a pixel of the second capturedimage, and (X1 ₍₃₎, Y1 ₍₃₎) represents the corresponding position on thefirst captured image. Note that a subscript (3) is used to indicate thata position of the third captured image is converted to a position withthe subscript (3).

The conversion parameters (θ3, T3x, T3y, S3) represent that a position(X3, Y3) on the third captured image is displaced with respect to acorresponding position (X1, Y1) on the first captured image by an angleθ3 due to a camera shake, enlarged (or reduced if S3<1) by a factor ofS3 due to movement of the digital camera 1 toward the subject, anddisplaced parallel to the subject by (T3x, T3y). In most cases, ofvarious displacement components of camera shake, a component in adirection parallel to the plane of the imaging device 4 has asignificant effect, but a displacement component in a direction from thedigital camera 1 to the subject is small enough to neglect it. That is,S2 can be approximated by S3=1.

Similarly, the first captured image used as the reference image issupplied to the motion detector 23-(k−1) from the frame memory 22-1, andthe k-th captured image used as the target image is supplied to themotion detector 23-(k−1) from the frame memory 22-k.

The motion detector 23-(k−1) detects the correspondence between theposition of each pixel (or each block when the image is divided into aplurality of blocks) of the k-th captured image and the position of thefirst captured image by determining conversion parameters (θk, Tkx, Tky,Sk) of equation (1) representing the positional relationship between thefirst captured image and the second captured image, where θk denotes arotation angle, Sk denotes a scale, and (Tkx, Tky) denotes a paralleldisplacement. The resultant conversion parameters (θk, Tkx, Tky, Sk) aresupplied to the calculation circuit 24. $\begin{matrix}{\begin{pmatrix}{X1}_{(k)} \\{Y1}_{(k)}\end{pmatrix} = {{{{Sk}\begin{pmatrix}{\cos\left( {\theta\quad k} \right)} & {- {\sin\left( {\theta\quad k} \right)}} \\{\sin\left( {\theta\quad k} \right)} & {\cos\left( {\theta\quad k} \right)}\end{pmatrix}}\begin{pmatrix}{Xk} \\{Yk}\end{pmatrix}} + {\begin{pmatrix}{Tkx} \\{Tky}\end{pmatrix}\left( {k = {4\quad{to}\quad N}} \right)}}} & (3)\end{matrix}$

Equation (3) is also in the form of an affine transformation, in which(Xk, Yk) represents the position of a pixel of the second capturedimage, and (X1(k), Y1(k)) represents the corresponding position on thefirst captured image. Note that a subscript (k) is used to indicate thata position of the second captured image is converted to a position withthe subscript (k).

The conversion parameters (θk, Tkx, Tky, Sk) represent that a position(Xk, Yk) on the k-th captured image is displaced with respect to acorresponding position (X1, Y1) on the first captured image by an angleθk due to a camera shake, enlarged (or reduced if Sk<1) by a factor ofSk due to movement of the digital camera 1 toward the subject, anddisplaced parallel to the subject by (Tkx, Tky). In most cases, ofvarious displacement components of camera shake, a component in adirection parallel to the plane of the imaging device 4 has asignificant effect, but a displacement component in a direction from thedigital camera 1 to the subject is small enough to neglect it. That is,Sk can be approximated by Sk=1.

Instead of determining the conversion parameters (θk, Tkx, Tky, Sk) fromthe positional relationship of the k-th captured image with respect tothe first captured image in the above-described manner, the conversionparameters (θk, Tkx, Tky, Sk) may be mechanically determined from asignal output from an acceleration sensor disposed on the digital camera1.

N captured images are supplied to the calculation circuit 24 from theframe memories 22-1 to 22-N, and the conversion parameters (θk, Tkx,Tky, Sk) representing the positional relationship between the firstcaptured image and the k-th captured image are supplied to thecalculation circuit 24 from the motion detectors 23-1 to 23-(N−1).

The calculation circuit 24 produces a displacement-compensated sharpimage signal (including a G signal component, R signal component, and Bsignal component) by performing a correct image estimation process,which will be described later, based on the positional relationships ofthe second to N-th captured images supplied from the motion detectors23-1 to 23-(N−1) with respect to the first captured image, and thecalculation circuit 24 supplies resultant displacement-compensated imagesignal to the D/A converter 9 or the codec 12. In each of the N capturedimages supplied from the A/D converter 6 to the signal processor 7, eachpixel has one of G, R, and B signals. On the other hand, in thecompensated image signal produced by the calculation circuit 24, eachpixel has all of G, R, and B signals (data).

Under the control of the CPU 15, the controller 25 controls variousparts of the signal processor 7, such as the frame memories 22-1 to22-N, the motion detectors 23-1 to 23-(N−1), the calculation circuit 24,etc. Instead of the controller 25, the CPU 15 (FIG. 1) may control partsof the signal processor 7, such as the frame memories 22-1 to 22-N, themotion detectors 23-1 to 23-(N−1), and calculation circuit 24. In thiscase, the controller 25 is not necessary.

In the single-plane sensor using the Bayer array, the number of pixelsfor R or B signals is less than the number of pixels for the G signal.This can cause the R signal or the B signal in the image output from thesignal processor 7 to have a greater error than an error the G signalhas. In such a case, a lowpass filter, which limits only ahigh-frequency component of the color signal without limiting theluminance signal, is disposed at a stage following the calculationcircuit 24 thereby removing noise.

If the calculation circuit 24 is designed to be capable of processingimages at a high speed on a real-time basis while the images aresequentially captured, the processing can be performed using a lessnumber of frame memories 22-1 to 22-N and a less number of motiondetectors 23-1 to 23-(N−1), and thus the signal processor 7 can beconstructed in a simpler form with less complexity.

Now, a correct image estimation process performed by the signalprocessor 7 according to a first embodiment of the present invention isdescribed.

Although not shown in the figures, an on-chip lens is disposed directlyon each pixel (photosensing part of each pixel) of the imaging device 4.Each on-chip lens focuses all light originating from a subject andincident on the on-chip lens into one point. Therefore, each pixellocated immediately under its own on-chip lens receives the integral ofall light incident on the on-chip lens. This results in an improvementin sensitivity of each pixel.

Therefore, data (light intensity) received by each pixel of the imagingdevice 4 is not the light intensity of light originating from a subjectand incident on a point (that is, not point-sampled data) but theintegral of light originating from the subject and incident on a finitearea of a plane (photosensing plane).

In the first embodiment, the characteristic of the on-chip lens isprecisely formulated so that a sharp image (output image) is producedvia a correct image estimation process. Note that in the conventionaltechniques, it is assumed that data is point-sampled at each pixel.However, in an actual imaging device 4, as described above, each pixelreceives the integral of light incident on a finite area of a plane, andthus the assumption of point-sampling results in an error in estimation(restoration) of a correct image.

Now, for the first image stored in the frame memory 22-1, therelationship between the pixel value obtained at each pixel and lightincident via the on-chip lens is formulated.

For example, for a pixel G00 at a (1st, 1st) position in upper leftcorner shown in FIG. 3, the relationship between the pixel value Gobs(1,1, 1) of that pixel and the true green light intensity Lg(x, y) is givenby a following equation (4). $\begin{matrix}{{\int_{0}^{1}{\int_{0}^{1}{{{Lg}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Gobs}\left( {1,1,1} \right)}} & (4)\end{matrix}$

The pixel G00 in upper left corner of the photosensor 4 serving as thesingle-plane sensor is covered with a green filter such that only agreen component of incident light can reach the pixel G00. Because ofthe provision of the on-chip lens, light incident on a rectangularregion whose corners are at coordinates (0, 0), (0, 1), (1, 0), and(1, 1) in FIG. 3 reaches the pixel G00. That is, all of light incidenton the rectangular region whose corners are at coordinates (0, 0), (0,1), (1, 0), and (1, 1) is sensed by the pixel G00 located at the (1st,1st) position.

The left-hand part in equation (4) represents the integral of the truegreen light intensity Lg(x, y) (incident) at a position (x, y) in thecoordinate system of the first captured image taken over a rectangularrange whose corners are located at (0, 0), (0, 1), (1, 0), and (1, 1),that is, over a range defined by 0≦x<1 and 0≦y<1. The right-hand part ofequation (4) represents the pixel value Gobs(1, 1, 1) obtained(observed) by the pixel at the (1st, 1st) position of the first image.Thus, equation (4) represents that the integral of the true green lightintensity Lg(x, y) incident on the light sensing plane of the pixel G00taken over the light sensing plane of the pixel G00 is equal to thepixel value Gobs(1, 1, 1) observed by the pixel G00 for the first image.

Depending on the characteristic of the on-chip lens, light originatingfrom a subject and being incident on a peripheral area (close to edges)of each pixel is not focused. In such a case, the range over which theintegral of the left-hand part of equation (4) is calculated is slightlyreduced. For example, the integral is calculated over a range whosecorners are at (0.1, 0.1), (0.1, 0.9), (0.9, 0.1), and (0.9, 0.9).

In many digital cameras, to avoid false color, which is a drawback of asingle-plane sensor, an optical lowpass filter is disposed on eachpixel. In this configuration, each pixel receives light incident on anarea slightly greater than a rectangular area of the pixel. In such acase, the rectangular range over which the integral of the left-handpart of equation (4) is calculated is slightly expanded.

The discussion will be continued by taking as an example a (2nd, 1st)pixel R00 located immediately adjacent to the right of the pixel G00 inFIG. 3.

For the pixel R01 at the (2nd, 1st) position, the relationship betweenthe pixel value Robs(1, 2, 1) of that pixel and the true green lightintensity Lg(x, y) is given by a following equation (5). $\begin{matrix}{{\int_{0}^{1}{\int_{1}^{2}{{{Lr}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Robs}\left( {1,2,1} \right)}} & (5)\end{matrix}$

The pixel R01 in upper left corner of the photosensor 4 serving as thesingle-plane sensor is covered with a red filter such that only a redcomponent of incident light can reach the pixel R01. Because of theprovision of the on-chip lens, light incident on a rectangular regionwhose corners are at coordinates (1, 0), (1, 1), (2, 0), and (2, 1) inFIG. 3 reaches the pixel R01. That is, all of light incident on therectangular region whose corners are at coordinates (1, 0), (1, 1), (2,0), and (2, 1) is sensed by the pixel R01 located at the (2nd, 1st)position.

The left-hand part in equation (5) represents the integral of the truered light intensity Lr(x, y) (incident) at a position (x, y) in thecoordinate system of the first captured image over a rectangular rangewhose corners are located at (1, 0), (1, 1), (2, 0), and (2, 1), thatis, over a range defined by 1≦x<2 and 0≦y<1. The right-hand part ofequation (5) represents the pixel value Robs(1, 2, 1) obtained(observed) by the pixel at the (2nd, 1st) position of the first image.Thus, equation (5) represents that the integral of the true red lightintensity Lr(x, y) incident on the light sensing plane of the pixel R01over the light sensing plane of the pixel R01 is equal to the pixelvalue Robs(1, 2, 1) observed by the pixel R01 for the first image.

Depending on the characteristic of the on-chip lens, light originatingfrom a subject and being incident on an a peripheral area (close toedges) of each pixel is not focused. In such a case, the range overwhich the integral of the left-hand part of equation (5) is calculatedis slightly reduced. For example, the integral is calculated over arange whose corners are at (1.1, 0.1), (1.1, 0.9), (1.9, 0.1), and (1.9,0.9).

In many digital cameras, to avoid false color, which is a drawback of asingle-plane sensor, an optical lowpass filter is disposed on eachpixel. In this configuration, each pixel receives light incident on anarea slightly greater than a rectangular area of the pixel. In such acase, the rectangular range over which the integral of the left-handpart of equation (5) is calculated is slightly expanded.

Similar equations are defined for the other pixels of the first capturedimage.

That is, for a pixel G(jg−1)(ig−1)at a (ig-th, jg-th) position shown inFIG. 3, the relationship between the pixel value Gobs(1, ig, jg) of thatpixel and the true green light intensity Lg(x, y) is given by afollowing equation (6). $\begin{matrix}{{\int_{{jg} - 1}^{jg}{\int_{{ig} - 1}^{ig}{{{Lg}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Gobs}\left( {1,{ig},{jg}} \right)}} & (6)\end{matrix}$

Equation (6) represents that the pixel value Gobs(1, ig, jg) is equal tothe integral of the true green light intensity Lg(x, y) over arectangular range whose corners are located at (ig−1, jg−1), (ig−1, jg),(ig, jg−1) and (ig, jg) in the coordinate system shown in FIG. 3, thatis, over a range defined by ig−1≦x<ig and jg−1≦y<jg.

For a pixel R(jr−1) (ir−1)at a (ir-th, jr-th) position shown in FIG. 3,the relationship between the pixel value Robs(1, ir, jr) of that pixeland the true red light intensity Lr(x, y) is given by a followingequation (7).∫_(jr−1) ^(jr)∫_(ir−1) ^(ir)Lr(x, y)dxdy=Robs(1, ir, jr)  (7)

Equation (7) represents that the pixel value Robs(1, ir, jr) is equal tothe integral of the true red light intensity Lr(x, y) over a rectangularrange whose corners are located at (ir−1, jr−1), (ir−1, jr), (ir, jr−1),and (ir, jr) in the coordinate system shown in FIG. 3, that is, over arange defined by ir−1≦x<ir and jr−1≦y<jr.

For a pixel B(jb−1)(ib−1)at a (ib-th, jb-th) position shown in FIG. 3,the relationship between the pixel value Bobs(1, ib, jb) of that pixeland the true blue light intensity Lb(x, y) is given by a followingequation (8). $\begin{matrix}{{\int_{{jb} - 1}^{jb}{\int_{{ib} - 1}^{ib}{{{Lb}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Bobs}\left( {1,{ib},{jb}} \right)}} & (8)\end{matrix}$

Equation (8) represents that the pixel value Bobs(1, ib, jb) is equal tothe integral of the true blue light intensity Lb(x, y) over arectangular range whose corners are located at (ib−1, jb−1), (ib−1, jb),(ib, jb−1), and (ib, jb) in the coordinate system shown in FIG. 3, thatis, over a range defined by ib−1≦x<ib and jb−1≦y<jb.

In practice, as described earlier, the image signal (the observed pixelvalue) includes noise E, which is gained up by a factor of Mk. If noisecomponents E×Mk are taken into account, formulas (6), (7), and (8) arerespectively rewritten into formulas (9), (10), and (11) indicatingconstraints that light intensity integrals should satisfy.$\begin{matrix}{{{{\int_{{jg} - 1}^{jg}{\int_{{ig} - 1}^{ig}{{{Lg}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} - {{Gobs}\left( {1,{ig},{jg}} \right)}}} \leq {{M1} \times E}} & (9) \\{{{{\int_{{jr} - 1}^{jr}{\int_{{ir} - 1}^{ir}{{{Lr}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} - {{Robs}\left( {1,{ir},{jr}} \right)}}} \leq {{M1} \times E}} & (10) \\{{{{\int_{{jb} - 1}^{jb}{\int_{{ib} - 1}^{ib}{{{Lb}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Bobs}\left( {1,{ib},{jb}} \right)}}} \leq {{M1} \times E}} & (11)\end{matrix}$where |x| denotes the absolute value of x.

Formula (9) represents that the difference between the integral of lightintensity Lg(x, y) and the observed pixel value Gobs(1, ig, jg) is equalto or less than a maximum possible value M1×E of noise. Inequalities(10) and (11) represent similar constraints.

In the following description, the k-th captured image will also bereferred to simply as the k-th image.

Now, for the second image stored in the frame memory 22-2, as for thefirst image, the relationship between the pixel value obtained(observed) at each pixel and light incident via the on-chip lens isrepresented in the form of a mathematical expression.

The coordinate system of the second and further following capturedimages is converted to the coordinate system of the first captured imagebased on the conversion parameters (θk, Tkx, Tky, Sk)(k=2 to N) detectedby the motion detector 23.

For a pixel G00 at a (1st, 1st) position of the second image, therelationship between the pixel value Gobs(2, 1, 1) of that pixel and thetrue green light intensity Lg(x, y) is given by a following equation(12). $\begin{matrix}{{\int_{0_{(2)}}^{1_{(2)}}{\int_{0_{(2)}}^{1_{(2)}}{{{Lg}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} = {{Gobs}\left( {2,1,1} \right)}} & (12)\end{matrix}$

The pixel G00 in upper left corner of the photosensor 4 is covered′witha green filter such that only a green component of incident light canreach the pixel G00. Because of the provision of the on-chip lens, lightincident on a rectangular region whose corners are at coordinates (0,0), (0, 1), (1, 0), and (1, 1) in the coordinate system of the secondimage reaches the pixel G00.

If positions (0, 0), (0, 1), (1, 0), and (1, 1) in the coordinate systemof the second image are converted to positions in the coordinate systemof the first image based on the conversion parameters (θ2, T2x, T2y,S2), the resultant positions are given as (0₍₂₎, 0₍₂₎), (0₍₂₎, 1₍₂₎),(1₍₂₎, 0₍₂₎), and (1₍₂₎, 1₍₂₎). That is, all of light incident on therectangular region whose corners are at coordinates (0₍₂₎, 0₍₂₎), (0₍₂₎,1₍₂₎), (1₍₂₎, 0₍₂₎) and (1₍₂₎, 1₍₂₎) is sensed by the pixel G00 locatedat the (1st, 1st) position. Hereinafter, when a position (x, y) of thesecond image is converted into a position in the coordinate system ofthe first image, a resultant position will be referred to as P(2, x, y).

The left-hand part in equation (12) represents the integral of the truegreen light intensity Lg(x, y) over a rectangular region whose cornersare at P(2, 0, 0), P(2, 0, 1), P(2, 1, 0), and P(2, 1, 1), that is, overa rectangular range whose corners are located at coordinates (0₍₂₎,0₍₂₎), (0₍₂₎, 1₍₂₎), (1₍₂₎, 0₍₂₎), and (1₍₂₎, 1₍₂₎) in the coordinatesystem of the first image. The right-hand part of equation (12)represents the pixel value Gobs(2, 1, 1) obtained at the pixel at the(1st, 1st) position of the second image. Equation (12) represents thatthe pixel value Gobs(2, 1, 1) observed at the pixel G00 of the secondimage is equal to the integral of the true green light intensity Lg(x,y) incident on a rectangular region whose corners are located at (0₍₂₎,0₍₂₎), (0₍₂₎, 1₍₂₎), (1₍₂₎, 0₍₂₎), and (1₍₂₎, 1₍₂₎) in the coordinatesystem of the first image. In equation (12), Jjdxdy represents theintegral taken over a rectangular region whose corners are at (0₍₂₎,0₍₂₎), (0₍₂₎, 1₍₂₎), (1₍₂₎, 0 ₍₂₎), and (1₍₂₎, 1₍₂₎). In equation (13)and other similar equations appearing in the following description,ifdxdy will be also used to represent a similar integral.

Depending on the characteristic of the on-chip lens, light originatingfrom a subject and being incident on an a peripheral area (close toedges) of each pixel is not focused. In such a case, the range overwhich the integral in the left-hand part of equation (12) is taken isslightly shrunk as in the first image.

In many digital cameras, to avoid false color, which is a drawback of asingle-plane sensor, an optical lowpass filter is disposed on eachpixel. In this configuration, each pixel receives light incident on anarea slightly greater than a rectangular area of the pixel. In such acase, the rectangular range over which the integral of the left-handpart of equation (12) is calculated is slightly expanded.

The discussion will be continued by taking as an example the (2nd, 1st)pixel R01 located immediately adjacent to the right of the pixel G00.

For the pixel R01 at the (2nd, 1st) position of the second image, therelationship between the pixel value Robs(2, 2, 1) of that pixel and thetrue red light intensity Lr(x, y) is given by a following equation (13).$\begin{matrix}{{\int_{0_{(2)}}^{1_{(2)}}{\int_{1_{(2)}}^{2_{(2)}}{{{Lr}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} = {{Robs}\left( {2,2,1} \right)}} & (13)\end{matrix}$

The pixel R01 in upper left corner of the photosensor 4 is covered witha red filter such that only a red component of incident light can reachthe pixel R01. Because of the provision of the on-chip lens, lightincident on a rectangular region whose corners are at coordinates (1,0), (1, 1), (2, 0), and (2, 1) in the coordinate system of the secondimage reaches the pixel R01.

If positions (1, 0), (1, 1), (2, 0), and (2, 1) in the coordinate systemof the second image are converted to positions in the coordinate systemof the first image based on the conversion parameters (θ2, T2x, T2y,S2), the resultant positions are given as (1₍₂₎, 0₍₂₎), (1₍₂₎, 1₍₂₎),(2₍₂₎, 0₍₂₎), and (2₍₂₎, 1₍₂₎). That is, all of light incident on therectangular region whose corners are at coordinates (1₍₂₎, 0₍₂₎), (1₍₂₎,1₍₂₎), (2₍₂₎, 0₍₂₎), and (2₍₂₎ 1 ₍₂₎) is sensed by the pixel R01 locatedat the (2nd, 1st) position.

The left-hand part in equation (13) represents the integral of the truered light intensity Lr(x, y) over a rectangular region whose corners areat P(2, 1, 0), P(2, 1, 1), P(2, 2, 0), and P(2, 2, 1), that is, over arectangular range whose corners are located at coordinates (1₍₂₎, 0₍₂₎),(1₍₂₎, 1₍₂₎), (2₍₂₎, 0₍₂₎), and (2₍₂₎, 1₍₂₎) in the coordinate system ofthe first image. The right-hand part of equation (13) represents thepixel value Robs(2, 2, 1) obtained at the pixel at the (2nd, 1st)position of the second image. Equation (13) represents that the pixelvalue Robs(2, 2, 1) observed at the pixel R01 of the second image isequal to the integral of the true green light intensity Lr(x, y)incident on a rectangular region whose corners are located at (1₍₂₎,0₍₂₎), (1₍₂₎, 1₍₂₎), (2₍₂₎, 1₍₂₎), and (2₍₂₎, 1₍₂₎) in the coordinatesystem of the first image.

Depending on the characteristic of the on-chip lens, light originatingfrom a subject and being incident on an a peripheral area (close toedges) of each pixel is not focused. In such a case, the range overwhich the integral in the left-hand part of equation (13) is taken isslightly.

In many digital cameras, to avoid false color, which is a drawback of asingle-plane sensor, an optical lowpass filter is disposed on eachpixel. In this configuration, each pixel receives light incident on anarea slightly greater than a rectangular area of the pixel. In such acase, the rectangular range over which the integral of the left-handpart of equation (13) is calculated is slightly expanded.

Similar equations are defined for the other pixels of the secondcaptured image.

That is, for a pixel G(jg−1)(ig−1)at a (ig-th, jg-th) position of thesecond image, the relationship between the pixel value Gobs(2, ig, jg)of that pixel and the true green light intensity Lg(x, y) is given by afollowing equation (14). $\begin{matrix}{{\int_{{jg} - 1_{(2)}}^{{jg}_{(2)}}{\int_{{ig} - 1_{(2)}}^{{ig}_{(2)}}{{{Lg}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Gobs}\left( {2,{ig},{jg}} \right)}} & (14)\end{matrix}$

Equation (14) represents that the pixel value Gobs(2, ig, jg) is equalto the integral of the true green light intensity Lg(x, y) over arectangular range whose corners are located at (ig−1, jg−1), (ig−1, jg),(ig, jg−1), and (ig, jg) in the coordinate system of the second image,that is, over a rectangular range whose corners are located at (ig−1₍₂₎,jg−1₍₂₎), (ig−1₍₂₎, jg₍₂₎), (ig₍₂₎, jg−1₍₂₎), and (ig₍₂₎, jg₍₂₎) in thecoordinate system of the first image.

That is, for a pixel R(jr−1)(ir−1)at a (ir-th, jr-th) position of thesecond image, the relationship between the pixel value Robs(2, ir, jr)of that pixel and the true red light intensity Lr(x, y) is given by afollowing equation (15). $\begin{matrix}{{\int_{{jg} - 1_{(2)}}^{{jg}_{(2)}}{\int_{{ir} - 1_{(2)}}^{{ir}_{(2)}}{{{Lr}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Robs}\left( {2,{ir},{jr}} \right)}} & (15)\end{matrix}$

Equation (15) represents that the pixel value Robs(2, ir, jr) is equalto the integral of the true red light intensity Lr(x, y) over arectangular range whose corners are located at (ir−1, jr−1), (ir−1, jr),(ir, jr−1), and (ir, jr) in the coordinate system of the second image,that is, over a rectangular range whose corners are located at (ir−1₍₂₎,jr−1₍₂₎), (ir−1₍₂₎, jr₍₂₎), (ir₍₂₎, jr−1₍₂₎), and (ir₍₂₎, jr₍₂₎) in thecoordinate system of the first image.

That is, for a pixel B(jb−1)(ib−1)at a (ib-th, jb-th) position of thesecond image, the relationship between the pixel value Bobs(2, ib, jb)of that pixel and the true red light intensity Lb(x, y) is given by afollowing equation (16). $\begin{matrix}{{\int_{{jb} - 1_{(2)}}^{{jb}_{(2)}}{\int_{{ib} - 1_{(2)}}^{{ib}_{(2)}}{{{Lb}\left( {x,y} \right)}\quad{\mathbb{d}x}\quad{\mathbb{d}y}}}} = {{Bobs}\left( {2,{ib},{jb}} \right)}} & (16)\end{matrix}$

Equation (16) represents that the pixel value Bobs(2, ib, jb) is equalto the integral of the true blue light intensity Lb(x, y) over arectangular range whose corners are located at (ib−1, jb−1), (ib−1, jb),(ib, jb−1), and (ib, jb) in the coordinate system of the second image,that is, over a rectangular range whose corners are located at (ib−1₍₂₎,jb−1₍₂₎), (ib−1₍₂₎, jb₍₂₎), (ib₍₂₎, jb−1₍₂₎), and (ib₍₂₎, jb₍₂₎) in thecoordinate system of the first image.

In practice, as described earlier, the image signal (the observed pixelvalue) includes noise E, which is gained up by a factor of Mk. If noisecomponents E×Mk are taken into account, formulas (14), (15), and (16)are respectively rewritten into formulas (17), (18), and (19) indicatingconstraints that light intensity integrals should satisfy.$\begin{matrix}{{{{\int_{{jg} - 1_{(2)}}^{{jg}_{(2)}}{\int_{{ig} - 1_{(2)}}^{{ig}_{(2)}}{{{Lg}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Gobs}\left( {2,{ig},{jg}} \right)}}} \leq {{M2} \times E}} & (17) \\{{{{\int_{{jr} - 1_{(2)}}^{{jr}_{(2)}}{\int_{{ir} - 1_{(2)}}^{{ir}_{(2)}}{{{Lr}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Robs}\left( {2,{ir},{jr}} \right)}}} \leq {{M2} \times E}} & (18) \\{{{{\int_{{jb} - 1_{(2)}}^{{jb}_{(2)}}{\int_{{ib} - 1_{(2)}}^{{ib}_{(2)}}{{{Lb}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Bobs}\left( {2,{ib},{jb}} \right)}}} \leq {{M2} \times E}} & (19)\end{matrix}$where |x| denotes the absolute value of x.

Formula (17) represents that the difference between the integral oflight intensity Lg(x, y) and the observed pixel value Gobs(2, ig, jg) isequal to or less than a maximum possible value M2×E of noise.Inequalities (18) and (19) represent similar constrains.

Now, referring to FIG. 5, the relationship between a position (x, y) ofthe second image and a position P(2, x, y) obtained by converting theposition (x, y) into coordinate system of the first image.

A pixel 31 in a rectangular region whose four corners are at (i−1, j−1),(I−1, j), (i, 1−1), and (i, j) in the coordinate system of the secondimage is shown on the left-hand side of FIG. 5.

The pixel 31 on the left-hand side is converted to a pixel 31′ in thecoordinate system of the first image, as shown on the right-hand side ofFIG. 5. Note that light of the same part of a subject (for example, alandscape scene) falls on the pixel 31 on the left-hand side of FIG. 5and the pixel 31′ on the right-hand side of FIG. 5.

The pixel value of each pixel of second and following captured images isdetermined as the integral of light in a particular region in the firstimage employed as the reference image. By properly setting theintegration region, the first to N-th captured images havingdisplacement are aligned into a correct position (with reference to thefirst captured image).

A position (i−1, j−1) in the coordinate system of the second image isconverted to a position P(2, i−1, j−1) in the coordinate system of thefirst image as a result of the conversion based on the conversionparameters (θ2, T2x, T2y, S2). A position (i−1, j) in the coordinatesystem of the second image is converted to a position P(2, i−1, j) inthe coordinate system of the first image as a result of the conversionbased on the conversion parameters (θ2, T2x, T2y, S2). Similarly,positions (i, j−1) and (i, j) in the coordinate system of the secondimage are converted to positions P(2, i, j−1) and P(2, i, j) in thecoordinate system of the first image as a result of the conversion basedon the conversion parameters (θ2, T2x, T2y, S2). Hereinafter, the Xcoordinate axis and the Y coordinate axis in the coordinate system ofthe first image will be simply referred to as X and Y, respectively.

As with the second image, positions (i−1, j−1), (i−1, j), (i, j−1) and(i, j) in the coordinate system of the k-th (k=3 to N) image areconverted to position P(k, i−1, j−1), P(k, i−1, j), P(k, i, j−1) andP(k, i, j) in the coordinate system of the first image as a result ofthe conversion based on the conversion parameters (θk, Tkx, Tky, Sk).

In the k-th image (k=3 to N), if noise E with a magnitude gained up by afactor Mk is taken into account, formulas (20) and (22) similar toformulas (17) and (19) indicating constraint imposed on the integral oflight density are obtained. $\begin{matrix}{{{{\int_{{jg} - 1_{(k)}}^{{jg}_{(k)}}{\int_{{ig} - 1_{(k)}}^{{ig}_{(k)}}{{{Lg}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Gobs}\left( {k,{ig},{jg}} \right)}}} \leq {{Mk} \times E}} & (20)\end{matrix}$

Formula (20) represents that the pixel value Gobs(k, ig, jg) obtained bya pixel located at a (ig-th, jg-th) position of the k-th image is equalto the integral of the true green light intensity Lg(x, y) taken over arectangular range whose corners are located at P(k, ig−1, ig−1), P(k,ig−1, ig), P(k, ig, jg−1), and P(k, ig, jg), that is, over a rectangularrange whose corners are located at (ig−1_((k)), jg−1_((k))),(ig−1_((k)), 1_((k))), (ig_((k)), jig−1_((k))), and (ig_((k)), jg_((k)))in the coordinate system of the first image, with a maximum possibleerror equal to noise E with a magnitude gained up by a factor Mk. Thatis, the (absolute value of) difference between the integral of truegreen light intensity Lg(x, y) and the observed pixel value Gobs(k, ig,jg) is equal to or less than a maximum possible value of noise. Notethat in formula (20), lxi denotes the absolute value of x.$\begin{matrix}{{{{\int_{{jr} - 1_{(k)}}^{{jr}_{(k)}}{\int_{{ir} - 1_{(k)}}^{{ir}_{(k)}}{{{Lr}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Robs}\left( {k,{ir},{jr}} \right)}}} \leq {{Mk} \times E}} & (21)\end{matrix}$

Formula (21) represents that the pixel value Robs(k, ir, jr) obtained bya pixel located at a (ir-th, jr-th) position of the k-th image is equalto the integral of the true red light intensity Lr(x, y) taken over arectangular range whose corners are located at P(k, ir−1, jr−1), P(k,ir−1, jr), P(k, ir, jr−1), and P(k, ir, jr), that is, over a rectangularrange whose corners are located at (ir−1_((k)), jr−1_((k))),(ir−1_((k)), jr(k)), (ir(k), jr−1_((k))), and (ir(k), jr(k) in thecoordinate system of the first image, with a maximum possible errorequal to noise E with a magnitude gained up by a factor Mk. That is, the(absolute value of) difference between the integral of true red lightintensity Lr(x, y) and the observed pixel value Robs(k, ir, jr) is equalto or less than a maximum possible value of noise. Note that in formula(21), |x| denotes the absolute value of x. $\begin{matrix}{{{{\int_{{jb} - 1_{(k)}}^{{jb}_{(k)}}{\int_{{ib} - 1_{(k)}}^{{ib}_{(k)}}{{{Lb}\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}} - {{Bobs}\left( {k,{ib},{jb}} \right)}}} \leq {{Mk} \times E}} & (22)\end{matrix}$

Formula (22) represents that the pixel value Bobs(k, ib, jb) obtained bya pixel located at a (ib-th, jb-th) position of the k-th image is equalto the integral of the true blue light intensity Lb(x, y) taken over arectangular range whose corners are located at P(k, ib−1, jb−1), P(k,ib−11, jb), P(k, ib, jb−1), and P(k, ib, jb), that is, over arectangular range whose corners are located at (ib−1_((k)), jb−1_((k))),(ib−1_((k)), jb_((k))), (ib (k), jb−1_((k))), and (ib (k), jb_((k)) inthe coordinate system of the first image, with a maximum possible errorequal to noise E with a magnitude gained up by a factor Mk. That is, the(absolute value of) difference between the integral of true blue lightintensity Lb(x, y) and the observed pixel value Bobs(k, ib, jb) is equalto or less than a maximum possible value of noise. Note that in formula(22), |x| denotes the absolute value of x.

Now, referring to a flow chart shown in FIG. 6, a correct imageestimation process in step S4 shown in FIG. 2 according to a firstembodiment of the present invention is described.

First, in step Sl1, the calculation circuit 24 produces a lightintensity integral formula in the form (9) from the pixel value Gobs(1,ig, jg) of each pixel of the first image for each (ig, jg), a lightintensity integral formula in the form (10) from Robs(1, ir, jr)for each(ir, jr), and a light intensity integral formula in the form (11) fromBobs(1, ib, jb) for each (ib, jb). Thereafter, the process proceeds tostep S12.

In step S12, the calculation circuit 24 produces a light intensityintegral formula in the form (17) from the pixel value Gobs(2, ig, jg)of each pixel of the second image for each (ig, jg), a light intensityintegral formula in the form (18) from Robs(2, ir, jr)for each (ir, jr),and a light intensity integral formula in the form (19) from Bobs(2, ib,jb) for each (ib, jb). Thereafter, the process proceeds to step S13. Inthe above process, the calculation circuit 24 converts positions of thesecond image to corresponding position of the first image by using theconversion parameters (θ2, T2x, T2y, S2) supplied from the motiondetector 23-1.

In step S13, the calculation circuit 24 produces a light intensityintegral formula in the form (20) from the pixel value Gobs(k, ig, jg)of each pixel of the k-th (k=3 to N) image for each (ig, jg), a lightintensity integral formula in the form (21) from Robs(k, ir, jr)for each(ir, jr), and a light intensity integral formula in the form (22) fromBobs(k, ib, jb) for each (ib, jb). Thereafter, the process proceeds tostep S14. In the above process, the calculation circuit 24 convertspositions of the k-th image to corresponding position of the first imageby using the conversion parameters (θk, Tkx, Tky, Sk) supplied from themotion detector 23-(k−1).

In step S14, the calculation circuit 24 estimates the true green lightintensity Lg(x, y), the true red light intensity Lr(x, y) and the trueblue light intensity Lb(x, y) by solving formulas associated with thelight density integral constraints (9), (10), (11), (17), (18), (19),(20), (21), and (22) produced in steps S11 to S13. If these values havebeen determined, the process is exited. The obtained values of the truegreen light intensity Lg(x, y), the true red light intensity Lr(x, y)and the true blue light intensity Lb(x, y) (estimated values thereof)are supplied as resultant image signals (a G signal, a R signal, and Bsignal) representing a sharp image to the D/A converter 9 or the codec12.

In this step S14, when the formulas associated with the light intensityintegral constraint are solved to determine the true green lightintensity Lg(x, y), the true red light intensity Lr(x, y) and true theblue light intensity Lb(x, y), a method of determining image data thatsatisfies a plurality of constraints is used. A specific example of sucha method is a POCS (Projection Onto Convex Sets) method. In the POCSmethod, a maximum-likelihood solution that satisfies given constraintsis estimated by repeatedly performing projection onto convex sets. Adescription of the details of this method may be found, for example, ina paper “Image Restoration by the Method of Convex Projections: part 1theory” (D. C. Youla and H. Webb, IEEE Trans. Med. Image., vol. 1, No.2, pp 81-94, October, 1982). Japanese Unexamined Patent ApplicationPublication No. 8-263639 discloses a technique of producinghigh-resolution still image from a plurality of low-resolution motionimages by using the POCS method. In this technique disclosed in JapaneseUnexamined Patent Application Publication No. 8-263639, motioncompensation is performed using motion vectors, and identificationaccuracy is equal to one or ½ pixels. The number of pixels used toestimate a most-likelihood pixel value of each pixel is fixed dependingon the identification accuracy. In contrast, in the present embodiment,motion compensation is performed based on a semi-analog process, and thenumber of pixels used to estimate a most-likelihood pixel value of eachpixel is adaptively changed depending on the state associated with thedisplacement due to the camera shake.

Now, referring to a flow chart shown in FIG. 7, a correct imageestimation processing step S4 shown in FIG. 2 according to the firstembodiment of the present invention is further described.

First, in step S21, the calculation circuit 24 sets a variable kindicating the count of the number of images stored in the calculationcircuit 24 to 1. Thereafter, the process proceeds to step S22.

In step S22, the calculation circuit 24 produces a light intensityintegral formula in the form (20) from the pixel value Gobs(k, ig, jg)of each pixel of the k-th image for each (ig, jg). Thereafter, theprocess proceeds to step S23.

In step S23, the calculation circuit 24 produces a light intensityintegral formula in the form (21) from the pixel value Robs(k, ir, jr)of each pixel of the k-th image for each (ir, jr). Thereafter, theprocess proceeds to step S24.

In step S24, the calculation circuit 24 produces a light intensityintegral formula in the form (22) from the pixel value Bobs(k, ib, jb)of each pixel of the k-th image for each (ib, jb). Thereafter, theprocess proceeds to step S25.

When the inequalities associated with the light intensity integral areproduced in steps S22 to S24 described above, the conversion parameterssupplied from the motion detectors 23-1 to 23-(N−1) are used asrequired.

In step S25, the calculation circuit 24 determines whether the internalvariable k is equal to the number of images N supplied from the framememories 22-1 to 22-N. If it is determined that the variable k is notequal to the number of images N, the process proceeds to step S26 inwhich the calculation circuit 24 increments the variable k by 1.Thereafter, the calculation circuit 24 returns the processing flow tostep S22 and repeats the process from S22 to S25.

On the other hand, if it is determined that the variable k is equal tothe number of images N, the process proceeds to step S27 in which thecalculation circuit 24 estimates the true green light intensity Lg(x,y), the true red light intensity Lr(x, y) and the true blue lightintensity Lb(x, y) by solving inequalities (20), (21), and (22) producedin steps S22 to S24. If these values have been determined, the processis completed. The obtained values of the green light intensity Lg(x, y),the red light intensity Lr(x, y) and the blue light intensity Lb(x, y)that are likely to be true values are supplied as resultant imagesignals (a G signal, a R signal, and B signal) representing a sharpimage to the D/A converter 9 or the codec 12.

The true green light intensity Lg(x, y), the true red light intensityLr(x, y), and the true blue light intensity Lb(x, y) obtained herein areanalog signals. That is, Lg(x, y), Lr(x, y), and Lb(x, y) are functionsof (x, y) wherein, as described earlier, variables×and y are notintegers but real numbers that can include decimal fractions. Ingeneral, each decimal fraction has two or three digits in binary,although the number of digits depends on the precision of the apparatus.Because Lg(x, y), Lr(x, y), and Lb(x, y) are functions of real numbers(x, y), resampling is performed at integral numbers of (x, y) asrequired, and the resultant digital image signal is supplied to the D/Aconverter 9 or the codec 12.

In a case in which a subject moves during a period in which N images aretaken at a high shutter speed, the true green light intensity Lg(x, y),the true red light intensity Lr(x, y), and the true blue light intensityLb(x, y) of a moving part of the subject change with time. In such acase, the method described above does not provide a correct solution.

Thus, when the subject includes a moving part, an exception handlingprocess is performed. In the exception handling process, simplesuperposition is performed. More specifically, from Bayer-array imagedata of each N images (having only one of R, G, and B signals at eachpixel), three signals consisting of R, G, and B signals are restored foreach pixel by means of a demosacing process. After the demosacingprocess, alignment is performed by rotating, increasing or reducing insize, and/or parallel displacing N images such that the N images arebrought to correct positions, and the N images are superimposedtogether. The demosacing process may be performed using any knowntechnique.

In the first embodiment, as described above, the processing is performedtaking into account the characteristic of the on-chip lens disposeddirectly on each pixel, and thus a sharp displacement-compensated imagecan be produced.

Now, a correct image estimation process performed by the signalprocessor 7 according to a second embodiment of the present invention isdescribed.

In the second embodiment, in addition to the constraint associated withthe light intensity integral given by formulas (20) to (21) for k=1 toN, a constraint associated with color correlation among R, G, and Bsignals is imposed on the process of determining the true green lightLg(x, y), the true red light intensity, and the blue light intensityLb(x, y).

In a local area of an image, there is a color correlation among colorsof the true green light intensity Lg(x, y), the true red light intensityLr(x, y), and the true blue light intensity Lb(x, y) corresponding tolight originating from a subject and being incident on the imagingdevice 4. By imposing the color correlation in addition to theconstraint associated with the light intensity integral given informulas (20) to (22), it becomes possible to obtain a better solution,that is, a sharper image more likely to be a correct image.

Referring to FIGS. 8 and 9, a specific method of determining a colorcorrelation constraint is described below. In the following descriptionreferring to FIGS. 8 and 9, the correlation between green and red colorswill be discussed.

By way of example, the discussion will be made for a green pixelG(jg−1)(ig−1) at a (ig-th, jg-th) position of a k′-th image shown in alower left part of FIG. 8 a red pixel G(jg−1)(ig−1) at a (ir-th, jr-th)position of a k″-th-image shown in a lower right part of FIG. 8.

The calculation circuit 24 converts positions of the green pixelG(jg−1)(ig−1) of the k′-th image and the red pixel R(jr−1)(ir−1) of thek″-th image into positions in the coordinate system of the first imageon the upper side of FIG. 8 by using conversion parameters (θk′, Tk′x,Tk′y, Sk′) and (0k11, Tk′1x, Tk″y, Sk″) in a similar manner to thataccording to the first embodiment. Note that ke and k″ can be equalto 1. When k′ or kl′ is equal to 1, the conversion parameters are givenas (θ1, T1x, T1y, S1)=(0, 0, 0, 1).

The calculation circuit 24 then calculates the distance between thegreen pixel G(jg−1)(ig−1) of the k′-th image at the position convertedinto the coordinate system of the first image and the red pixelR(jr−1)(ir−1) of the k″-th image at the position converted into thecoordinate system of the first image. Furthermore, the calculationcircuit 24 determines whether these two pixels can be regarded aslocating at the same position by determining whether the calculateddistance is less than a maximum allowable value (threshold value) delta(set to, for example, 0.25 times the pixel size).

When positions of pixels G(jg−1)(ig−1) and R(jr−1) (ir−1) arerespectively given as positions (ig, jg) and (ir, jr), if the position(ig, jg) of the pixel G(jg−1)(ig−1) in the coordinate system of thek′-th image is denoted as point G(c), the position (ir, jr) of the pixelR(jg−1) (ig−1) in the coordinate system of the k″-th image is denoted aspoint R(c), the position (ig, jg) of the pixel G(jg−1)(ig−1) in thecoordinate system of the first image is denoted as G_(c(k′)), and theposition (ir, jr) of the pixel R(jg−1)(ig−1) in the coordinate system ofthe first image is denoted as RC(k), then a constraint requiring thatthe distance between the point G_(c(k′)) and point R_(c(k′)) should bewithin an allowable value delta is given by formula (23) shown below.$\begin{matrix}{{{Dis}\left( {G_{c{(k^{\prime})}},R_{c{(k^{\prime\prime})}}} \right\rbrack} = {{{Dis}\left\lbrack {{{{{Sk}^{\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ig} \\{jg}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime}x} \\{{Tk}^{\prime}y}\end{pmatrix}},{{{{Sk}^{\prime\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ir} \\{jr}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime\prime}x} \\{{Tk}^{\prime\prime}y}\end{pmatrix}}} \right\rbrack} \leq {delta}}} & (23)\end{matrix}$

Formula (23) is referred to as a distance constraint formula. In formula(23), Dis[G_(c(k′)), RC(k..)] denotes the distance between point Gc(k′)and point Rc(k″). The positions of point Gc(ka)) and point Rc(k..) areobtained by performing the affine transformation on positions (ig, jg)and (ir, jr) using conversion parameters (θk′, Tk′x, Tk′y, Sk′) and(θk″, Tk″x, Tk″y, Sk″) according to an equation similar to equations (1)to (3).

The calculation circuit 24 determines whether there are a green pixelG(jg−1)(ig−1) of the k′-th image and a red pixel R(jr−1)(ir−1) of thek″-th image at positions that can be regarded as being the same withinan allowable margin delta in a small region (x±dX, y±dY) centered at aposition (x, y) in the coordinate system of the first image, that is, ina rectangular region whose corners are at (x−dX, y−dY), (x−dX, y+dY),(x+dX, y−dY), and (x+dX, y+dY), wherein dX and dY are values that definethe region in the vicinity of the position (x, y) and they are set to beequal to, for example, 2 times the size of a pixel in the X directionand Y direction, respectively.

In other words, the calculation circuit 24 determines (k′, ig, jg) and(k′1, ir, jr) that satisfy formula (23) within the small region (x+dX,y±dY) centered at the position (x, y) in the coordinate system of thefirst image, that is, in the rectangular region whose corners are at(x−dX, y−dY), (x−dX, y+dY), (x+dX, y−dY), and (x+dX, y+dY).

The calculation circuit 24 then acquires pixel values Gobs(k′, ig, jg)and Robs(k″, ir, jr) respectively corresponding to (k′, ig, jg) and (k″,ir, jr) determined.

The calculation circuit 24 determines (k′, ig, jg) and (k″, ir, jr) thatsatisfy formula (23) for all possible combinations of k′ and k″ withinthe range form 1 to N.

In general, a plurality of combinations of (k′, ig, jg) and (k″, ir, jr)is detected, and the calculation circuit 24 plots pixel values Gobs(k′,ig, jg) and Robs(k″, ir, jr) corresponding to the detected (k′, ig, jg)and (k″, ir, jr) in the GR space in which the horizontal axis representsthe G signal (Gobs(k′, ig, jg)) and the vertical axis represents the Rsignal (Robs(k″, ir, jr)), as shown in FIG. 9.

FIG. 9 is a schematic diagram showing pixel values Gobs(k′, ig, jg) andRobs(k″, ir, jr) regarded as locating at same positions and plotted inthe GR space.

In FIG. 9, each x-like mark indicates a pair of pixel values Gobs(k′,ig, jg) and Robs(k″, ir, jr) corresponding to (k′, ig, jg) and (k″, ir,jr) detected by the calculation circuit 24.

The plots shown in FIG. 9 indicate that there is a correlation betweenthe true green light intensity Lg(x, y) and the true red light intensityLr(x, y) to be determined in the vicinity of the position (x, y).

Thus, in the second embodiment, in addition to constraints associatedwith the light intensity integral given by formulas (20) to (22) in thefirst embodiment, the correlation between green and red colors shown inFIG. 9 is added as a constrain.

The calculation circuit 24 performs principal component analysis for theplurality of points plotted in the GR space shown in FIG. 9, whereineach plotted point corresponds to a pair of pixel values Gobs(k′, ig,jg) and Robs(k″, ir, jr) that the calculation circuit 24 has detected assatisfying the distance constraint (23).

The calculation circuit 24 then determines the variance of a component(for example, a second principal component) in a direction perpendicularto a direction (denoted by a great arrow in FIG. 9) of a principalcomponent (first principal component) obtained as a result of theprincipal component analysis. Furthermore, the calculation circuit 24expresses a color correlation constraint in the form of a mathematicalexpression asserting that, for the position (x, y), a pointcorresponding to true green light intensity Lg(x, y) and true red lightintensity Lr(x, y) should be within a band extending in the GR space inthe direction of principal component and having a width equal to thevariance of the component in the direction perpendicular to thedirection of the principal component.

The color correlation constraint between green and blue is alsointroduced in a similar manner as described below.

As with the green and red pixels shown in FIG. 8, the calculationcircuit 24 converts positions of a green pixel G(jg−1)(ig−1) of thek′-th image and a blue pixel B(jb−1) (jb−1) of the k′ ′-th image intopositions in the coordinate system of the first image by usingconversion parameters (θk′, Tk′x, Tk′y, Sk′) and (θk′″, Tk′″x, Tk″y,Sk′″) in a similar manner to that according to the first embodiment.

The calculation circuit 24 then calculates the distance between thegreen pixel G(jg−1)(ig−1) of the k′-th image at the position convertedinto the coordinate system of the first image and the blue pixelV(jb−1)(ib−1) of the kll′-th image at the position converted into thecoordinate system of the first image. Furthermore, the calculationcircuit 24 determines whether these two pixels can be regarded aslocating at the same position by determining whether the calculateddistance is less than a maximum allowable value (threshold value) deltain a similar manner to FIG. 8.

When positions of pixels G(jg−1)(ig−1) and B(jb−1) (ib−1) arerespectively given as positions (ig, jg) and (ib, jb), if the position(ig, jg) of the pixel G(jg−1)(ig−1) in the coordinate system of thek′-th image is denoted as G(c), the position (ib, jb) of the pixelB(jb−1)(ib−1) in the coordinate system of the k′″-th image is denoted aspoint B(c), the position (ig, jg) of the pixel G(jg−1)(ig−1) in thecoordinate system of the first image is denoted as G_(c(k′)), and theposition (ib, jb) of the pixel B(jb−1) (ib−1) in the coordinate systemof the first image is denoted as BC(k . . . ), then a constraintrequiring that the distance between point G_(c(k′)) and point BC(k . . .) should be within the allowable value delta is given by formula (24)shown below. $\begin{matrix}{{{Dis}\left( {G_{c{(k^{\prime})}},B_{c{(k^{\prime\prime\prime})}}} \right\rbrack} = {{{Dis}\left\lbrack {{{{{Sk}^{\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ig} \\{jg}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime}x} \\{{Tk}^{\prime}y}\end{pmatrix}},{{{{Sk}^{\prime\prime\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime\prime\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime\prime\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime\prime\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime\prime\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ib} \\{jb}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime\prime\prime}x} \\{{Tk}^{\prime\prime\prime}y}\end{pmatrix}}} \right\rbrack} \leq {delta}}} & (24)\end{matrix}$

Formula (24) is referred to as a distance constraint formula. In formula(24), Dis[GC(k.), BC(k . . . )] denotes the distance between pointGC(k.) and point BC(k .). The positions of point G_(c(k′))) and pointBC(k . . . ) are obtained by performing the affine transformation onpositions (ig, jg) and (ib, jb) using conversion parameters (θk′, Tk′x,Tk′y, Sk′) and (θk′″, Tk″tx, Tk′″y, Sk′″) according to an equationsimilar to equations (1) to (3).

The calculation circuit 24 determines whether there are a green pixelG(jg−1)(ig−1) of the k′-th image and a blue pixel B(jb−1)(ib−1) of thek′″-th image at positions that can be regarded as being the same withinan allowable margin delta in a small region (x±dX, y±dY) centered at aposition (x, y) in the coordinate system of the first image, that is, ina rectangular region whose corners are at (x−dX, y−dY), (x−dX, y+dY),(x+dX, y−dY), and (x+dX, y+dY) wherein dX and dY are values that definethe region in the vicinity of the position (x, y) and they are set to beequal to, for example, 2 times the size of a pixel in the X directionand Y direction, respectively.

In other words, the calculation circuit 24 determines (k′, ig, jg) and(k″, ib, jb) that satisfy formula (24) within the small region (x±dX,y±dY) centered at the position (x, y) in the coordinate system of thefirst image, that is, in the rectangular region whose corners are at(x−dX, y−dY), (x−dX, y+dY), (x+dX, y−dY), and (x+dX, y+dY).

The calculation circuit 24 then acquires pixel values Gobs(k′, ig, jg)and Bobs(k′″, ib, jb) respectively corresponding to (kl, ig, jg) and(k′″, ib, jb) determined.

The calculation circuit 24 determines (k′, ig, jg) and (k″, ib, jb) thatsatisfy formula (24) for all possible combinations of k′ and k′,′ withinthe range form 1 to N.

In general, a plurality of combinations of (k′, ig, jg) and (k′″, ib,jb) is detected, and the calculation circuit 24 plots pixel valuesGobs(k′, ig, jg) and Bobs(k′″, ib, jb) corresponding to the detected(k′, ig, jg) and (k′″, ib, jb) in the GB space in which the horizontalaxis represents the G signal (Gobs(k′, ig, jg)) and the vertical axisrepresents the B signal (Bobs(k′″, ib, jb)).

Thus, in the second embodiment, in addition to constraints associatedwith the light intensity integral given by formulas (20) to (22) in thefirst embodiment, the correlation between green and blue colors similarto the correlation between green and red shown in FIG. 9 is added as aconstrain.

The calculation circuit 24 performs principal component analysis for theplurality of points plotted in the GB space, wherein each plotted pointcorresponds to a pair of pixel values Gobs(k′, ig, jg) and Bobs(k′″, ib,jb) detected by the calculation circuit 24.

The calculation circuit 24 then determines the variance of a component(for example, a second principal component) in a direction perpendicularto a direction of a principal component (first principal component)obtained as a result of the principal component analysis. Furthermore,the calculation circuit 24 expresses a color correlation constraint inthe form of a mathematical expression asserting that, for the position(x, y), a point corresponding to true green light intensity Lg(x, y) andtrue blue light intensity Lb(x, y) should be within a band extending inthe GB space in the direction of principal component and having a widthequal to the variance of the component in the direction perpendicular tothe direction of the principal component.

Thus, the true green light intensity Lg(x, y), the true red lightintensity Lr(x, y), and the true blue light intensity Lb(x, y) finallyobtained by the calculation circuit 24 are limited to those satisfyingthe constraint that a point corresponding to the true green lightintensity Lg(x, y) and the true red light intensity Lr(x, y) at theposition (x, y) exists within a band extending, in the GR space, in thedirection of the principal component and having a width equal to thevariance of the component perpendicular to the direction of theprincipal component, and satisfying the constraint that a pointcorresponding to the true green light intensity Lg(x, y) and the trueblue light intensity Lb(x, y) at the position (x, y) exists within aband extending, in the GB space, in the direction of the principalcomponent and having a width equal to the variance of the componentperpendicular to the direction of the principal component.

Although in the present embodiment, the constraint associated with thecolor correlation in the GR space and the constraint associated with thecolor correlation in the GB space, a similar constraint associated witha color correlation between the R signal and the B signal (in a RBspace) may also be employed in addition to the above constraints.

As in the first embodiment, for example, the POCS method may be used todetermine the true green light intensity Lg(x, y), the true redintensity Lr(x, y), and the true blue light intensity Lb(x, y) thatsatisfy both the constraint associated with the light intensityintegrals and the constraint associated with the color correlations.

The color correlation constraints may be imposed on all positions (x, y)or only particular positions such as lattice points at which×and y areequal to integers.

Now, referring to a flow chart shown in FIG. 10, a correct imageestimation process in step S4 shown in FIG. 2 according to a secondembodiment of the present invention is described.

Steps S31 to S33 are similar to steps S11 to S13 in the correct imageestimation process according to the first embodiment shown in FIG. 6.

That is, in step S31, the calculation circuit 24 produces a lightintensity integral formula in the form (9) from the pixel value Gobs(1,ig, jg) of each pixel of the first image for each (ig, jg), a lightintensity integral formula in the form (10) from Robs(1, ir, jr)for each(ir, jr), and a light intensity integral formula in the form (11) fromBobs(1, ib, jb) for each (ib, jb). Thereafter, the process proceeds tostep S32.

In step S32, the calculation circuit 24 produces a light intensityintegral formula in the form (17) from the pixel value Gobs(2, ig, jg)of each pixel of the second image for each (ig, jg), a light intensityintegral formula in the form (18) from Robs(2, ir, jr)for each (ir, jr),and a light intensity integral formula in the form (19) from Bobs(2, ib,jb) for each (ib, jb). Thereafter, the process proceeds to step S33. Inthe above process, the calculation circuit 24 converts positions of thesecond image to corresponding position of the first image by using theconversion parameters (θ2, T2x, T2y, S2) supplied from the motiondetector 23-1.

In step S33, the calculation circuit 24 produces a light intensityintegral formula in the form (20) from the pixel value Gobs(k, ig, jg)of each pixel of the k-th (k=3 to N) image for each (ig, jg), a lightintensity integral formula in the form (21) from Robs(k, ir, jr)for each(ir, jr), and a light intensity integral formula in the form (22) fromBobs(k, ib, jb) for each (ib, jb). Thereafter, the process proceeds tostep S34. In the above process, the calculation circuit 24 convertspositions of the k-th image to corresponding position of the first imageby using the conversion parameters (θk, Tkx, Tky, Sk) supplied from themotion detector 23-(k−1).

In step S34, for a position (x, y) of interest, the calculation circuit24 determines a pair of (k′, ig, jg) and (k″, ir, jr) that satisfy thedistance constraint represented in equality (23) in the vicinity of theposition (x, y). Thereafter, the process proceeds to step S35.

In step S35, the calculation circuit 24 plots, in the GR space, thepixel values Gobs(k′, ig, jg) and Robs(k″, ir, jr) at respectivepositions (k′, ig, jg) and (k″, ir, jr) determined in step S34 andperforms principal component analysis. The calculation circuit 24 thendetermines the variance in a direction perpendicular to a principaldirection determined as a result of the principal component analysis,and represents a color correlation constraint in the form of amathematical expression asserting that the G signal and the R signal ofthe same pixel exist within the range equal to the determined variance.Thereafter, the process proceeds from step S35 to step S36.

In step S36, for a position (x, y) of interest, the calculation circuit24 determines a pair of (k′, ig, jg) and (k′″, ib, jb) that satisfy thedistance constraint represented in equality (24) in the vicinity of theposition (x, y). Thereafter, the process proceeds to step S37.

In step S37, the calculation circuit 24 plots, in the GB space, thepixel values Gobs(k′, ig, jg) and Bobs(k′″, ib, jb) at respectivepositions (k′, ig, jg) and (k′″, ib, jb) determined in step S36 andperforms principal component analysis. The calculation circuit 24 thendetermines the variance in a direction perpendicular to a principaldirection determined as a result of the principal component analysis,and represents a color correlation constraint in the form of amathematical expression asserting that the G signal and the B signal ofthe same pixel exist within the range equal to the determined variance.Thereafter, the process proceeds from step S37 to step S38.

In step S38, the calculation circuit 24 determines whether the colorcorrelation constraint has been determined in the form of a mathematicalexpression for all positions (x, y). If it is determined in step S38that the color correlation constraint has not been determined for allpositions (x, y), the process returns to step S34. In step S34, thecalculation circuit 24 selects a position (x, y) for which the colorcorrelation constraint has not been determined, and repeats the processfrom S34 to S38.

On the other hand, in the case in which it is determined in step S38that the color correlation constraint has been determined for allpositions (x, y), the process proceeds to step S39. In step S39, thecalculation circuit 24 calculates the true green light density Lg(x, y),the true red light density Lr(x, y), and the true blue light densityLb(x, y) that satisfy all constraints determined in steps S31, S32, S33,S35, and S37. Thereafter, the processing flow returns from the currentprocess. In the above calculation, the calculation circuit 24 determinesthe true green light intensity Lg(x, y), the true red light intensityLr(x, y), and the true blue light intensity Lb(x, y) that satisfy allthe light intensity constraint formulas determined in steps S31 to S33and the color correlation constraints determined in steps S35 and S37.

Now, referring to flow charts shown in FIG. 11 and 12, a correct imageestimation process in step S4 shown in FIG. 2 according to the secondembodiment of the present invention is further described.

Steps S51 to S56 are similar to steps S21 to S26 in the correct imageestimation process according to the first embodiment shown in FIG. 7.

That is, in step S51, the calculation circuit 24 sets a variable kindicating the count of the number of images stored in the calculationcircuit 24 to 1. Thereafter, the process proceeds to step S52.

In step S52, the calculation circuit 24 produces a light intensityintegral formula in the form (20) from the pixel value Gobs(k, ig, jg)of each pixel of the k-th image for each (ig, jg). Thereafter, theprocess proceeds to step S53.

In step S53, the calculation circuit 24 produces a light intensityintegral formula in the form (21) from the pixel value Robs(k, ir, jr)of each pixel of the k-th image for each (ir, jr). Thereafter, theprocess proceeds to step S54.

In step S54, the calculation circuit 24 produces a light intensityintegral formula in the form (22)′from the pixel value Bobs(k, ib, jb)of each pixel of the k-th image for each (ib, jb). Thereafter, theprocess proceeds to step S55.

When the inequalities associated with the light intensity integral areproduced in steps S52 to S54 described above, the conversion parameterssupplied from the motion detectors 23-1 to 23-(N−1) are used asrequired.

In step S55, the calculation circuit 24 determines whether the internalvariable k is equal to the number of images N supplied from the framememories 22-1 to 22-N. If it is determined that the variable k is notequal to the number of images N, the process proceeds to step S56 inwhich the calculation circuit 24 increments the variable k by 1.Thereafter, the calculation circuit 24 returns the processing flow tostep S52 and repeats the process from S52 to S56.

On the other hand, if it is determined in step S55 that the variable kis equal to the number of images N, the process proceeds to step S57.Steps S57 to S62 are similar to steps S34 to S39 shown in FIG. 10.

That is, in step S57, for a position (x, y) of interest, the calculationcircuit 24 determines a pair of (k′, ig, jg) and (k″, ir, jr) thatsatisfy the distance constraint represented in equality (23) in thevicinity of the position (x, y). Thereafter, the process proceeds tostep S58.

In step S58, the calculation circuit 24 plots, in the GR space, thepixel values Gobs(k′, ig, jg) and Robs(k′1, ir, jr) at respectivepositions (k′, ig, jg) and (k″, ir, jr) determined in step S57 andperforms principal component analysis. The calculation circuit 24 thendetermines the variance in a direction perpendicular to a principaldirection determined as a result of the principal component analysis,and represents a color correlation constraint in the form of amathematical expression asserting that the G signal and the R signal ofthe same pixel exist within the range equal to the determined variance.Thereafter, the process proceeds from step S58 to step S59.

In step S59, for a position (x, y) of interest, the calculation circuit24 determines a pair of (k′, ig, jg) and (k′″, ib, jb) that satisfy thedistance constraint represented in equality (24) in the vicinity of theposition (x, y). Thereafter, the process proceeds to step S60.

In step S60, the calculation circuit 24 plots, in the GB space, thepixel values Gobs(k′, ig, jg) and Bobs(k′″, ib, jb) at respectivepositions (k′, ig, jg) and (k′″, ib, jb) determined in step S59 andperforms principal component analysis. The calculation circuit 24 thendetermines the variance in a direction perpendicular to a principaldirection determined as a result of the principal component analysis,and represents a color correlation constraint in the form of amathematical expression asserting that the G signal and the B signal ofthe same pixel exist within the range equal to the determined variance.Thereafter, the process proceeds from step S60 to step S61.

In step S61, the calculation circuit 24 determines whether the colorcorrelation constraint has been determined in the form of a mathematicalexpression for all positions (x, y). If it is determined in step S61that the color correlation constraint has not been determined for allpositions (x, y), the process returns to step S57. In step S57, thecalculation circuit 24 selects a position (x, y) for which the colorcorrelation constraint has not been determined, and repeats the processfrom S57 to S61.

On the other hand, in the case in which it is determined in step S61that the color correlation constraint has been determined for allpositions (x, y), the process proceeds to step S62. In step S62, thecalculation circuit 24 calculates the true green light density Lg(x, y),the true red light density Lr(x, y), and the true blue light densityLb(x, y) that satisfy all constraints determined in steps S52, S53, S54,S58, and S60. Thereafter, the processing flow returns from the currentprocess. In the above calculation, the calculation circuit 24 determinesthe true green light intensity Lg(x, y), the true red light intensityLr(x, y), and the true blue light intensity Lb(x, y) that satisfy allthe light intensity constraint formulas determined in steps S52 to S54and the color correlation constraints determined in steps S58 and S60.

In the present embodiment, as described above, the constraint associatedwith correlation among the R signal, the G signal and the B signal isadded to the constraints associated with light intensity integrals givenby inequalities (20), (21), and (22) for k=1 to N in the firstembodiment, and the true green light intensity Lg(x, y), the true redlight intensity Lr(x, y), and the true blue light intensity Lb(x y)satisfying all above constraints. This makes it possible to obtain asharp image close to an ideal image.

Although in the example described above, the color correlationconstraints are defined in the 2-dimensional color spaces, the colorcorrelation constraint may be defined in a 3-dimensional color space.

Now, a correct image estimation process performed by the signalprocessor 7 according to a third embodiment of the present invention isdescribed.

In the first and second embodiments described above, the true greenlight intensity Lg(x, y), the true red light intensity Lr(x, y), and thetrue blue light intensity Lb(x, y) are estimated taking into account theaccurately formulated characteristic of the on-chip lens disposeddirection on each pixel of the imaging device 4, thereby obtaining asharp image.

In the third embodiment, the effects of the on-chip lens are ignored,and it is assumed that the intensity of light originating from a subjectand being incident on a point (for example, the barycenter of eachpixel) of each pixel is detected by that pixel of the imaging device 4(that is, point-sampled data is obtained by each pixel).

Furthermore, as will be described in detail later, a spring model isused in this third embodiment. This makes it possible to reduce thecomputational complexity required to determine the true green lightintensity Lg(x, y), the true red light intensity Lr(x, y), and the trueblue light intensity Lb(x, y), compared with the POCS method used in thefirst and second embodiments. The third embodiment also provides anadvantage that it is possible to produce a compensated image withsharpness similar to that achieved by the first or second embodiment andbetter than can be achieved by the conventional techniques.

In the third embodiment, as described above, data sensed by each pixelis regarded as point-sampled data. More specifically, at a pixel at a(i-th, j-th) position of the imaging device 4 shown in FIG. 3, data isregarded as being sensed at a central point (with coordinates (i−0.5,j−0.5)) of the pixel at the (i-th, j-th) position.

Therefore, for example, a pixel value Gobs(1, ig, jg) obtained as databy a pixel at a (ig-th, jg-th) position of a first image is green lightintensity Lg(ig−0.5, jg−0.5) sensed at a position (ig−0.5, jg−0.5) inthe coordinate system of the first image. Similarly, a pixel valueRobs(1, ir, jr) obtained as data by a pixel at a (ir-th, jr-th) positionof the first image is red light intensity Lr(ir−0.5, jr−0.5) sensed at aposition (ir−0.5, jr−0.5) in the coordinate system of the first image,and a pixel value Bobs(1, ib, jb) obtained as data by a pixel at a(ib-th, jb-th) position of the first image is blue light intensityLb(ib−0.5, jb−0.5) sensed at a position (ib−0.5, jb−0.5) in thecoordinate system of the first image.

FIG. 13 shows the first image stored in the frame memory 22-1.

In FIG. 13, as for the pixel G(jg−1)(ig−1) that senses a greencomponent, a pixel value Gobs(1, ig, jg) is observed at a positiondenoted by a solid circle. As for the pixel R(jr−1)(ir−1) that senses ared component, a pixel value Robs(1, ir, jr) is observed at a positiondenoted by a solid square. As for the pixel B(jb−1)(ib−1) that senses ablue component, a pixel value Bobs(1, ib, jb) is observed at a positiondenoted by a solid triangle. As described above, the pixel value of eachpixel of the first image is observed at the center (barycenter), asrepresented in the coordinate system of the first image, of each pixel.For example, in the case of a pixel at a (i-th, j-th) position, thepixel value is observed at a position (i−0.5, j−0.5).

FIG. 14 shows a second image stored in the frame memory 22-2.

In FIG. 14, as for the pixel G(jg−1)(ig−1) that senses a greencomponent, a pixel value Gobs(2, ig, jg) is observed at a positiondenoted by a solid circle. As for the pixel R(jr−1)(ir−1) that senses ared component, a pixel value Robs(2, ir, jr) is observed at a positiondenoted by a solid square. As for the pixel B(jb−1)(ib−1) that senses ablue component, a pixel value Bobs(2, ib, jb) is observed at a positiondenoted by a solid triangle. As described above, the pixel value of eachpixel of the second image is observed at the center (barycenter), asrepresented in the coordinate system of the second image, of each pixel.For example, in the case of a pixel at a (i-th, j-th) position, thepixel value is observed at a position (i−0.5, j−0.5).

As in the first and second embodiments, points on the second image areconverted to corresponding points in the coordinate system of the firstimage. More specifically, each point on the second image is converted toa corresponding point in the coordinate system of the first image byusing conversion parameters (θ2, T2x, T2y, S2) detected by the motiondetector 23-1.

Corresponding points in the coordinate system of the first imageconverted from points on the second image are also shown in FIG. 14, forpixel values Gobs(2, ig, jg), Robs(2, ir, jr), and Bobs(2, ib, jb).

That is, the pixel value Gobs(2, ig, jg) of a pixel G(jg−1)(ig−1) in thecoordinate system of the second image is green light intensity Lg(x, y)observed at a position ((ig−0.5)₍₂₎, (jg−0.5)₍₂₎) in the coordinatesystem of the first image obtained by converting a position (ig−0.5,jg−0.5) in the coordinate system of the second image using conversionparameters (θ2, T2x, T2y, S2) detected by the motion detector 23-1. InFIG. 14, the position ((ig−0.5)₍₂₎, (jg−0.5)₍₂₎) in the coordinatesystem of the first image where the pixel value Gobs(2, ig, jg) isindicated by an open circle.

The pixel value Robs(2, ir, jr) of a pixel R(jr−1)(ir−1) in thecoordinate system of the second image is red light intensity Lr(x, y)observed at a position ((ir−0.5)₍₂₎, (jr−0.5)₍₂₎) in the coordinatesystem of the first image obtained by converting a position (ir−0.5,jr−0.5) in the coordinate system of the second image using conversionparameters (θ2, T2x, T2y, S2) detected by the motion detector 23-1. InFIG. 14, the position ((ir−0.5)₍₂₎, (jr−0.5)₍₂₎) in the coordinatesystem of the first image where the pixel value Robs(2, ir, jr) isindicated by an open square.

The pixel value Bobs(2, ib, jb) of a pixel B(jb−1)(ib−1) in thecoordinate system of the second image is blue light intensity Lb(x, y)observed at a position ((ib−0.5)₍₂₎, (jb−0.5)₍₂₎) in the coordinatesystem of the first image obtained by converting a position (ib−0.5,jb−0.5) in the coordinate system of the second image using conversionparameters (θ2, T2x, T2y, S2) detected by the motion detector 23-1. InFIG. 14, the position ((ib−0.5)₍₂₎, (jb−0.5)₍₂₎) in the coordinatesystem of the first image where the pixel value Bobs(2, ib, jb) isindicated by an open triangle.

FIG. 15 shows a third image stored in the frame memory 22-3.

In FIG. 15, as for the pixel G(jg−1)(ig−1) that senses a greencomponent, a pixel value Gobs(3, ig, jg) is observed at a positiondenoted by a solid circle. As for the pixel R(jr−1)(ir−1) that senses ared component, a pixel value Robs(3, ir, jr) is observed at a positiondenoted by a solid square. As for the pixel B(jb−1)(ib−1) that senses ablue component, a pixel value Bobs(3, ib, jb) is observed at a positiondenoted by a solid triangle. As described above, the pixel value of eachpixel of the third image is observed at the center (barycenter), asrepresented in the coordinate system of the third image, of each pixel.For example, in the case of a pixel at a (i-th, j-th) position, thepixel value is observed at a position (i−0.5, j−0.5).

As in the first and second embodiments, points on the third image areconverted to corresponding points in the coordinate system of the firstimage. More specifically, each point on the third image is converted toa corresponding point in the coordinate system of the first image byusing conversion parameters (θ3, T3x, T3y, S3) detected by the motiondetector 23-2.

Corresponding points in the coordinate system of the first imageconverted from points on the third image are also shown in FIG. 15, forpixel values Gobs(3, ig, jg), Robs(3, ir, jr), and Bobs(3, ib, jb).

That is, the pixel value Gobs(3, ig, jg) of a pixel G(jg−1)(ig−1) in thecoordinate system of the third image is green light intensity Lg(x, y)observed at a position ((ig−0.5)₍₃₎, (jg−0.5)₍₃₎) in the coordinatesystem of the first image obtained by converting a position (ig−0.5,jg−0.5) in the coordinate system of the third image using conversionparameters (θ3, T3x, T3y, S3) detected by the motion detector 23-2. InFIG. 15, the position ((ig−0.5)₍₃₎, (jg−0.5)₍₃₎) in the coordinatesystem of the first image where the pixel value Gobs(3, ig, jg) isindicated by an open circle.

The pixel value Robs(3, ir, jr) of a pixel R(jr−1)(ir−1) in thecoordinate system of the second image is red light intensity Lr(x, y)observed at a position ((ir−0.5)₍₃₎, (jr−0.5)₍₃₎) in the coordinatesystem of the first image obtained by converting a position (ir−0.5,jr−0.5) in the coordinate system of the third image using conversionparameters (θ3, T3x, T3y, S3) detected by the motion detector 23-2. InFIG. 15, the position ((ir−0.5)₍₃₎, (jr−0.5)₍₃₎) in the coordinatesystem of the first image where the pixel value Robs(3, ir, jr) isindicated by an open square.

The pixel value Bobs(3, ib, jb) of a pixel B(jb−1)(ib−1) in thecoordinate system of the third image is blue light intensity Lb(x, y)observed at a position ((ib−0.5)₍₃₎, (jb−0.5)₍₃₎) in the coordinatesystem of the first image obtained by converting a position (ib−0.5,jb−0.5) in the coordinate system of the third image using conversionparameters (θ3, T3x, T3y, S3) detected by the motion detector 23-2. InFIG. 15, the position ((ib−0.5)₍₃₎, (jb−0.5)₍₃₎) in the coordinatesystem of the first image where the pixel value Bobs(3, ib, jb) isindicated by an open triangle.

FIG. 16 shows positions, converted into the coordinate system of thefirst image, at which pixel values Gobs(1, ig, jg) to Gobs(N, ig, jg) ofrespective first to N-th images are observed at a pixel G(jg−1)(ig−1)that senses a green component.

In the example shown in FIG. 16, a pixel value Gobs(1, ig, jg) of thefirst image is observed at the center (pixel center) of a pixelG(jg−1)(ig−1) at a (ig-th, jg-th) position in the coordinate system ofthe first image. A pixel value Gobs(2, ig, jg) of the second image isobserved at a position converted into the coordinate system of the firstimage in an upper-left area of the pixel G(jg−1)(ig−1). A pixel valueGobs(3, ig, jg) of the third image is observed at a position convertedinto the coordinate system of the first image in a lower-left area ofthe pixel G(jg−1) (ig−1). A pixel value Gobs(4, ig, jg) of the fourthimage is observed at a position converted into the coordinate system ofthe first image in an upper-right area of the pixel G(jg−1)(ig−1). Notethat pixel values Gobs(k, ig, jg) (k=5 to N) of fifth to N-th images arenot shown in FIG. 16.

In this third embodiment, the calculation circuit 24 determines the truegreen light intensity Lg(ig−0.5, jg−0.5) at a position (ig−0.5, jg−0.5)in the coordinate system of the first image, based on pixel valuesGobs(k, ig, jg) (k=1 to N) of respective first to N-th images.

Herein, the center position (i−0.5, j−0.5) of a pixel at a (i-th, j-th)position in the coordinate system of the first image used as a referenceimage is rewritten as (I′, J′).

That is, I′=i−0.5 and J′=j−0.5. The true green light intensity Lg(I′,J′) to be determined can be represented by Lg(ig−0.5, jg−0.5) at thecenter position (ig−0.5, jg−0.5) of a pixel that senses a greencomponent. Similarly, the true red light intensity Lr(I′, J′) to bedetermined can be represented by Lr(ir−0.5, jr−0.5) at the centerposition (ir−0.5, jr−0.5) of a pixel that senses a red component, andthe true blue light intensity Lb(I′, J′) to be determined can berepresented by Lb(ib−0.5, jb−0.5) at the center position (ib−0.5,jb−0.5) of a pixel that senses a blue component.

FIG. 17 shows positions, as represented in the coordinate system of thefirst image, of the true red light intensity Lr(I′, J′), the true redlight intensity Lr(I′, J′), and the true blue light intensity Lb(I′, J′)to be determined by the calculation circuit 24 as image signals of asharp image.

That is, in FIG. 17, the center positions (I′, J′), as represented inthe coordinate system of the first image, of respective pixels of theimaging device 4 at which the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, J′), and the true blue light intensityLb(I′, J′) to be determined are indicated by solid circles.

Referring to FIGS. 18 to 23, the spring model is described taking, as anexample, a pixel that senses a green component. In FIG. 18 and followingfigures, the positions (I′, i′), as represented in the coordinate systemof the first image, of the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, J′), and the true blue light intensityLb(I′, J′) to be determined are indicated by solid circles, positions,as represented in the coordinate system of the first image, of observedpixel values Gobs(k, ig, jg), Robs(k, ir, jr), Bobs(k, ib, jb) (k=1 toN) are indicated by open circles. Hereinafter, the coordinate system ofthe first image used as the reference image will also be referred to asa reference coordinate system.

In FIG. 18, a position (x, y) of a point A indicates a position((ig−0.5)(k), (jg−0.5)(k)) in the reference coordinate system obtainedby converting a position (ig−0.5, jg−0.5) at which a pixel value Gobs(k,ig, jg) of a pixel at a (ig-th, jg-th) position of a k-th image wasobserved, by using conversion parameters (θk, Tkx, Tky, Sk) detected bythe motion detector 23-(k−1). When K=1, the conversion parameters areregarded as (θk, Tkx, Tky, Sk)=(0, 0, 0, 1), and thus (x, y)=(ig−0.5,jg−0.5). Note that the pixel value at point A(x, y) is Gobs(k, ig, jg).

For the given point A(x, y), integers I and J that satisfy I−0.2≦x<1+0.5and J−0.5≦y<J+0.5, respectively, are determined. Positions (I′, J′) atwhich to determine the true green light intensity Lg(I′, J′) are givenby four corners (I−0.5, J−0.5), (I−0.5, J+0.5), (I+0.5, J−0.5), and(I+0.5, J+0.5) of a rectangle (I+0.5, J±0.5).

The pixel value Gobs(k, ig, jg) at the point A(x, y) can be regarded asbeing nearly equal to true green light intensities Lg(I−0.5, J−0.5),Lg(I−0.5, J+0.5), Lg(I+0.5, J−0.5), and Lg(I+0.5, J+0.5) at respectivelypositions (I−0.5, J−0.5), (I−0.5, J+0.5), (I+0.5, J−0.5), and (I+0.5,J+0.5) close to the point A(x, y). That is, the true green lightintensities Lg(I−0.5, J−0.5), Lg(I−0.5, J+0.5), Lg(I+0.5, J−0.5), andLg(I+0.5, J+0.5) can be approximated by the pixel value Gobs(k, ig, jg)at the point A(x, y).

As described in the first and second embodiments, the pixel valueGobs(k, ig, jg) observed at the point A(x, y) includes an error (noise)Mk×E. Furthermore, each of the true green light intensities Lg(I−0.5,J−0.5), Lg(I−0.5, J+0.5), Lg(I+0.5, J−0.5), and Lg(I+0.5, J+0.5) atrespective positions (I−0.5, J−0.5), (I−0.5, J+0.5), (I+0.5, J−0.5), and(I+0.5, J+0.5) includes an additional error caused by approximating themby the pixel value Gobs(k, ig, jg) at the point A(x, y). (Hereinafter,such an error will be referred to as an approximation error).

The approximation of the true green light intensity, for example,Lg(I−0.5, J−0.5) at the position (I−0.5, J−0.5) by the pixel valueGobs(k, ig, jg) at the point A(x, y) can be represented by the springmodel using a spring.

FIG. 19 shows the concepts of the spring model by which the true greenlight intensity Lg(I−0.5, J−0.5) at the position (I−0.5, J−0.5) isapproximated by the pixel value Gobs(k, ig, jg) at the point A(x, y).

As shown in FIG. 19, one end of a spring BA1 is connected to the pixelvalue Gobs(k, ig, jg) and the other end of the spring BA1 is connectedto the true green light intensity Lg(I−0.5, J−0.5). In a particularstate shown in FIG. 19, the true green light intensity Lg(I−0.5, J−0.5)is being pulled toward the pixel value Gobs(k, ig, jg). In the springmodel shown in FIG. 19, the degree to which the true green lightintensity Lg(I−0.5, J−0.5) is pulled toward the pixel value Gobs(k, ig,jg) increases with the likelihood that the pixel value Gobs(k, ig, jg)at the point A(x, y) is the true green light intensity Lg(I−0.5, J−0.5).Conversely, the true green light intensity Lg(I−0.5, J−0.5) moves awayfrom the pixel value Gobs(k, ig, jg) when the likelihood is low.

The approximation error caused by approximating the true green lightintensity Lg(I−0.5, J−0.5) by the pixel value Gobs(k, ig, jg) increaseswith the distance between the point A(x, y) and the position (I−0.5,J−0.5). Thus, herein, a function F((x1, y1), (x2, y2)) that increaseswith the distance between a point (x1, y1) and a point (x2, y2) isintroduced. As for the function F((x1, y1), (x2, y2)), for example,F((x1, y1), (x2, y2))=√{(x1,−x2)²+(y1−y2)²} may be employed.

As described earlier, the pixel value Gobs(k, ig, jg) observed at thepoint A(x, y) includes an error (noise) Mk x E caused by noise.

The likelihood of equality between the true green light intensityLg(I−0.5, J−0.5) and the pixel value Gobs(k, ig, jg), that is, theapproximation accuracy of the true green light intensity Lg(I−0.5,J−0.5) approximated by the pixel value Gobs(k, ig, jg) decreases orincreases with increasing or decreasing approximation error and noise Ex Mk. Therefore, the approximation accuracy (likelihood) decreases with(in inverse proportion to) Mk×E×F((x, y), (I−0.5, J−0.5)). That is, thelikelihood that the true green light intensity Lg(I−0.5, J−0.5) is equalto the pixel value Gobs(k, ig, jg) at the point A(x, y) is high when thevalue of Mk×E×F((x, y), (I−0.5, J−0.5)) is large. Conversely, thelikelihood low is high when the value of Mk×E×F((x, y), (I−0.5, J−0.5))is small.

If the likelihood of equality of the true green light intensityLg(I−0.5, J−0.5) to the pixel value Gobs(k, ig, jg) is represented bythe spring constant (strength) of the spring BA1, then the springconstant is given, for example, as {√2−F((x, y), (I−0.5,J−0.5))}/(Mk×E). The denominator (Mk×E) is equal to Mk times the noiseE. Therefore, as noise increases, the strength of the spring BA1decreases and thus the force of pulling the true green light intensityLg(I−0.5, J−0.5) toward the pixel value Gobs(k, ig, jg) at the point Adecreases. The numerator {√2−F ((x, y), (I−0.5, J−0.5))} is equal to thevalue (difference) obtained by subtracting the distance between thepoint A and the point (I−0.5, J−0.5) from √(1²+1²)=√2, that is, themaximum possible value of the distance between two arbitrary pointswithin the rectangular region (I+0.5, J±0.5) shown in FIG. 18.Therefore, as the distance increases between the point A(x, y) at whichthe pixel value Gobs(k, ig, jg) is observed and the position (I−0.5,J−0.5) of the true green light intensity Lg(I−0.5, J−0.5) to bedetermined, V2- F((x, y), (I−0.5, J−0.5)) decreases and thus thestrength of the spring BA1 decreases. Thus, the force of the spring BA1to pull the true green light intensity Lg(I−0.5, J−0.5) toward the pixelvalue Gobs(k, ig, jg) at the point A decreases. Instead of {√2−F((x, y),(I−0.5, J−0.5))}, other functions such as the inverse of F((x, y),(I−0.5, J−0.5)) may be used as the numerator of the spring constant.

In the spring model, if the spring BA1 is assumed to have a naturallength of 0 (that is, the length is equal to 0 when no load is imposedon the spring), then the length (extension) of the spring BA1 along anaxis of light intensity (pixel value) can be written as I Gobs(k, ig,jg)−Lg(I−0.5, J−0.5)|, and the force exerted by the spring BA1 on thetrue green light intensity Lg(I−0.5, J−0.5) to pull it toward the pixelvalue Gobs(k, ig, jg) at the point A (x, y) can be written as the springconstant x the length of the spring={√2−F((x, y), (I−0.5,J−0.5))}/(Mk×E)}×|Gobs(k, ig, jg)−Lg(I−0.5, J−0.5)|.

FIG. 20 shows the concepts of the spring model of the relationshipbetween the pixel value Gobs(k, ig, jg) at the point A(x, y) and truegreen light intensities Lg(I−0.5, J−0.5), Lg(I−0.5, J+0.5), Lg(I+0.5,J−0.5), and Lg(I+0.5, J+0.5) at respective positions (I−0.5, J−0.5),(I−0.5, J+0.5), (I+0.5, J−0.5), and (I+0.5, J+0.5).

As with the spring model of the relationship between the pixel valueGobs(k, ig, jg) at the point A(x, y) and the true green light intensityLg(I−0.5, J−0.5) described above with reference to FIG. 19, therelationship of the true green light intensities Lg(I−0.5, J+0.5),Lg(I+0.5, J−0.5), and Lg(I+0.5, J+0.5) with the pixel value Gobs(k, ig,jg) at the point A(x, y) is also represented by the spring model.

As shown in FIG. 20, one end of a spring BA2 is connected to the pixelvalue Gobs(k, ig, jg) and the other end of the spring BA2 is connectedto the true green light intensity Lg(I−0.5, J+0.5), and the true greenlight intensity Lg(I−0.5, J+0.5) is being pulled toward the pixel valueGobs(k, ig, jg). In the spring model shown in FIG. 20, the degree towhich the true green light intensity Lg(I−0.5, J+0.5) is pulled towardthe pixel value Gobs(k, ig, jg) increases with the likelihood that thepixel value Gobs(k, ig, jg) at the point A(x, y) is the true green lightintensity Lg(I−0.5, J+0.5). Conversely, the true green light intensityLg(I−0.5, J+0.5) moves away from the pixel value Gobs(k, ig, jg) whenthe likelihood is low.

The approximation error caused by approximating the true green lightintensity Lg(I−0.5, J+0.5) by the pixel value Gobs(k, ig, jg) increaseswith the distance between the point A(x, y) and the position (I−0.5,J+0.5). Thus, herein, a function F((x1, y1), (x2, y2)) that increaseswith the distance between a point (x1, y1) and a point (x2, y2) isintroduced. As for the function F((x1, y1), (x2, y2)), for example,F((x1, y1), (x2, y2))=√{(x1,−x2)²+(y1−y2)²} may be employed.

As described earlier, the pixel value Gobs(k, ig, jg) observed at thepoint A(x, y) includes an error (noise) Mk×E caused by noise.

The likelihood of equality between the true green light intensityLg(I−0.5, J+0.5) and the pixel value Gobs(k, ig, jg), that is, theapproximation accuracy of the true green light intensity Lg(I−0.5,J+0.5) approximated by the pixel value Gobs(k, ig, jg) decreases orincreases with increasing or decreasing approximation error and noiseE×Mk. Therefore, the approximation accuracy (likelihood) decreases with(in inverse proportion to) Mk×E×F((x, y), (I−0.5, J+0.5)). That is, thelikelihood that the true green light intensity Lg(I−0.5, J+0.5) is equalto the pixel value Gobs(k, ig, jg) at the point A(x, y) is high when thevalue of Mk×E×F((x, y), (I−0.5, J+0.5)) is large. Conversely, thelikelihood low is high when the value of Mk×E×F((x, y), (I−0.5, J+0.5))is small.

If the likelihood of equality of the true green light intensityLg(I−0.5, J+0.5) to the pixel value Gobs(k, ig, jg) is represented bythe spring constant (strength) of the spring BA2, then the springconstant is given, for example, as {√2−F((x, y), (I−0.5,J+0.5))}/(Mk×E). The denominator (Mk×E) is equal to Mk times the noiseE. Therefore, as noise increases, the strength of the spring BA2decreases and thus the force of pulling the true green light intensityLg(I−0.5, J+0.5) toward the pixel value Gobs(k, ig, jg) at the point Adecreases. The numerator {√2−F((x, y), (I−0.5, J+0.5))} is equal to thevalue (difference) obtained by subtracting the-distance between thepoint A and the point (I−0.5, J+0.5) from √(1²+1²)=√2, that is, themaximum possible value of the distance between two arbitrary pointswithin the rectangular region (I±0.5, J+0.5) shown in FIG. 18.Therefore, as the distance increases between the point A(x, y) at whichthe pixel value Gobs(k, ig, jg) is observed and the position (I−0.5,J+0.5) of the true green light intensity Lg(I−0.5, J+0.5) to bedetermined, √2−F((x, y), (I−0.5, J+0.5)) decreases and thus the strengthof the spring BA2 decreases. Thus, the force of the spring BA2 to pullthe true green light intensity Lg(I−0.5, J+0.5) toward the pixel valueGobs(k, ig, jg) at the point A decreases. Instead of {√2−F((x, y),(I−0.5, J+0.5))}, other functions such as the inverse of F((x, y),(I−0.5, J+0.5)) may be used as the numerator of the spring constant.

In the spring model, if the spring BA2 is assumed to have a naturallength of 0 (that is, the length is equal to 0 when no load is imposedon the spring), then the length (extension) of the spring BA2 along anaxis of light intensity (pixel value) can be written as |Gobs(k, ig,jg)−Lg(I−0.5, J+0.5)|, and the force exerted by the spring BA2 on thetrue green light intensity Lg(I−0.5, J+0.5) to pull it toward the pixelvalue Gobs(k, ig, jg) at the point A (x, y) can be written as the springconstant x the length of the spring={√2−F((x, y), (I−0.5,J+0.5))}/(Mk×E)}×|Gobs(k, ig, jg)−Lg(I−0.5, J+0.5)|. Furthermore, asshown in FIG. 20, one end of a spring BA3 is connected to the pixelvalue Gobs(k, ig, jg) and the other end of the spring BA3 is connectedto the true green light intensity Lg(I+0.5, J−0.5), and the true greenlight intensity Lg(I+0.5, J−0.5) is being pulled toward the pixel valueGobs(k, ig, jg). In the spring model shown in FIG. 20, the degree towhich the true green light intensity Lg(I+0.5, J−0.5) is pulled towardthe pixel value Gobs(k, ig, jg) increases with the likelihood that thepixel value Gobs(k, ig, jg) at the point A(x, y) is the true green lightintensity Lg(I+0.5, J−0.5). Conversely, the true green light intensityLg(I+0.5, J−0.5) moves away from the pixel value Gobs(k, ig, jg) whenthe likelihood is low.

The approximation error caused by approximating the true green lightintensity Lg(I+0.5, J−0.5) by the pixel value Gobs(k, ig, jg) increaseswith the distance between the point A(x, y) and the position (I+0.5,J−0.5). Thus, herein, a function F((x1, y1), (x2, y2)) that increaseswith the distance between a point (x1, y1) and a point (x2, y2) isintroduced. As for the function F((x1, y1), (x2, y2)), for example,F((x1, y1), (x2, y2))=√{(x1,−x2)²+(y1−y2)²} may be employed.

As described earlier, the pixel value Gobs(k, ig, jg) observed at thepoint A(x, y) includes an error (noise) Mk×E caused by noise.

The likelihood of equality between the true green light intensityLg(I+0.5, J−0.5) and the pixel value Gobs(k, ig, jg), that is, theapproximation accuracy of the true green light intensity Lg(I+0.5,J−0.5) approximated by the pixel value Gobs(k, ig, jg) decreases orincreases with increasing or decreasing approximation error and noiseE×Mk. Therefore, the approximation accuracy (likelihood) decreases with(in inverse proportion to) Mk×E×F((x, y), (I+0.5, J−0.5)). That is, thelikelihood that the true green light intensity Lg(I+0.5, J−0.5) is equalto the pixel value Gobs(k, ig, jg) at the point A(x, y) is high when thevalue of Mk×E×F((x, y), (I+0.5, J−0.5)) is large. Conversely, thelikelihood low is high when the value of Mk×E×F((x, y), (I+0.5, J−0.5))is small.

If the likelihood of equality of the true green light intensityLg(I+0.5, J−0.5) to the pixel value Gobs(k, ig, jg) is represented bythe spring constant (strength) of the spring BA3, then the springconstant is given, for example, as {√2−F((x, y), (I+0.5,J−0.5))}/(Mk×E). The denominator (Mk×E) is equal to Mk times the noiseE. Therefore, as noise increases, the strength of the spring BA3decreases and thus the force of pulling the true green light intensityLg(I+0.5, J−0.5) toward the pixel value Gobs(k, ig, jg) at the point Adecreases. The numerator {√2−F((x, y), (I+0.5, J−0.5))} is equal to thevalue (difference) obtained by subtracting the distance between thepoint A and the point (I+0.5, J−0.5) from √(1²+1²)=√2, that is, themaximum possible value of the distance between two arbitrary pointswithin the rectangular region (I±0.5, J+0.5) shown in FIG. 18.Therefore, as the distance increases between the point A(x, y) at whichthe pixel value Gobs(k, ig, jg) is observed and the position (I+0.5,J−0.5) of the true green light intensity Lg(I+0.5, J−0.5) to bedetermined, √2−F((x, y), (I+0.5, J−0.5)) decreases and thus the strengthof the spring BA3 decreases. Thus, the force of the spring BA3 to pullthe true green light intensity Lg(I+0.5, J−0.5) toward the pixel valueGobs(k, ig, jg) at the point A decreases. Instead of {√2−F((x, y),(I+0.5, J−0.5))}, other functions such as the inverse of F((x, y),(I+0.5, J−0.5)) may be used as the numerator of the spring constant.

In the spring model, if the spring BA3 is assumed to have a naturallength of 0 (that is, the length is equal to 0 when no load is imposedon the spring), then the length (extension) of the spring BA3 along anaxis of light intensity (pixel value) can be written as |Gobs(k, ig,jg)−Lg(I+0.5, J−0.5)|, and the force exerted by the spring BA3 on thetrue green light intensity Lg(I+0.5, J−0.5) to pull it toward the pixelvalue Gobs(k, ig, jg) at the point A (x, y) can be written as the springconstant×the length of the spring={f2- F((x, y), (I+0.5,J−0.5))}/(Mk×E)}×|Gobs(k, ig, jg)−Lg(I+0.5, J−0.5)|.

Furthermore, in FIG. 20, one end of a spring BA4 is connected to thepixel value Gobs(k, ig, jg) and the other end of the spring BA4 isconnected to the true green light intensity Lg(I+0.5, J+0.5), and thetrue green light intensity Lg(I+0.5, J+0.5) is being pulled toward thepixel value Gobs(k, ig, jg). In the spring model shown in FIG. 20, thedegree to which the true green light intensity Lg(I+0.5, J+0.5) ispulled toward the pixel value Gobs(k, ig, jg) increases with thelikelihood that the pixel value Gobs(k, ig, jg) at the point A(x, y) isthe true green light intensity Lg(I+0.5, J+0.5). Conversely, the truegreen light intensity Lg(I+0.5, J+0.5) moves away from the pixel valueGobs(k, ig, jg) when the likelihood is low.

The approximation error caused by approximating the true green lightintensity Lg(I+0.5, J+0.5) by the pixel value Gobs(k, ig, jg) increaseswith the distance between the point A(x, y) and the position (I+0.5,J+0.5). Thus, herein, a function F((x1, y1), (x2, y2)) that increaseswith the distance between a point (x1, y1) and a point (x2, y2) isintroduced. As for the function F((x1, y1), (x2, y2)), for example,F((x1, y1), (x2, y2))=√{(x1,−x2)²+(y1−y2)²} may be employed.

As described earlier, the pixel value Gobs(k, ig, jg) observed at thepoint A(x, y) includes an error (noise) Mk x E caused by noise.

The likelihood of equality between the true green light intensityLg(I+0.5, J+0.5) and the pixel value Gobs(k, ig, jg), that is, theapproximation accuracy of the true green light intensity Lg(I+0.5,J+0.5) approximated by the pixel value Gobs(k, ig, jg) decreases orincreases with increasing or decreasing approximation error and noise Ex Mk. Therefore, the approximation accuracy (likelihood) decreases with(in inverse proportion to) Mk×E×F((x, y), (I+0.5, J+0.5)). That is, thelikelihood that the true green light intensity Lg(I+0.5, J+0.5) is equalto the pixel value Gobs(k, ig, jg) at the point A(x, y) is high when thevalue of Mk×E×F((x, y), (I+0.5, J+0.5)) is large. Conversely, thelikelihood low is high when the value of Mk×E×F((x, y), (I+0.5, J+0.5))is small.

If the likelihood of equality of the true green light intensityLg(I+0.5, J+0.5) to the pixel value Gobs(k, ig, jg) is represented bythe spring constant (strength) of the spring BA4, then the springconstant is given, for example, as {√2−F((x, y), (I+0.5,J+0.5))}/(Mk×E). The denominator (Mk×E) is equal to Mk times the noiseE. Therefore, as noise increases, the strength of the spring BA4decreases and thus the force of pulling the true green light intensityLg(I+0.5, J+0.5) toward the pixel value Gobs(k, ig, jg) at the point Adecreases. The numerator {√2−F((x, y), (I+0.5, J+0.5))} is equal to thevalue (difference) obtained by subtracting the distance between thepoint A and the point (I+0.5, J+0.5) from √(1²+1²)=√2, that is, themaximum possible value of the distance between two arbitrary pointswithin the rectangular region (I±0.5, J±0.5) shown in FIG. 18.Therefore, as the distance increases between the point A(x, y) at whichthe pixel value Gobs(k, ig, jg) is observed and the position (I+0.5,J+0.5) of the true green light intensity Lg(I+0.5, J+0.5) to bedetermined, √2−F((x, y), (I+0.5, J+0.5)) decreases and thus the strengthof the spring BA4 decreases. Thus, the force of the spring BA4 to pullthe true green light intensity Lg(I+0.5, J+0.5) toward the pixel valueGobs(k, ig, jg) at the point A decreases. Instead of {√2−F((x, y),(I+0.5, J+0.5))}, other functions such as the inverse of F((x, y),(I+0.5, J+0.5)) may be used as the numerator of the spring constant.

In the spring model, if the spring BA4 is assumed to have a naturallength of 0 (that is, the length is equal to 0 when no load is imposedon the spring), then the length (extension) of the spring BA4 along anaxis of light intensity (pixel value) can be written as |Gobs(k, ig,ig)−Lg(I+0.5, J+0.5)|, and the force exerted by the spring BA4 on thetrue green light intensity Lg(I+0.5, J+0.5) to pull it toward the pixelvalue Gobs(k, ig, jg) at the point A (x, y) can be written as the springconstant x the length of the spring={√2−F((x, y), (I+0.5,J+0.5))}/(Mk×E)}×|Gobs(k, ig, jg)−Lg(I+0.5, J+0.5)|.

In the process described above, for a given arbitrary position A(x, y)in the reference coordinate system, the spring model is introduced toexpress the relationship between the pixel value Gobs(k, ig, jg) at theposition A(x, y) and the true green light intensities Lg(I−0.5, J−0.5),Lg(I−0.5, J+0.5), Lg(I+0.5, J−0.5), and Lg(I+0.5, J+0.5) at positions inthe vicinity of the position (x, y), that is, at positions indicated byintegers I and J that satisfy inequalities I−0.5≦x<I+0.5 andJ−0.5≦y<J+0.5. Instead, for a given position (I′, J′) of interest in thereference coordinate system at which the center of a pixel is located,the relationship between the true green light intensity Lg(I′, J′) atthe position (I′, J′) and pixel values Gobs(k, ig, jg) observed in thevicinity of the position (I′, J′) may be expressed by a spring model, aswill be described below.

Note that (I′, J′) indicates the center position of a pixel, and thusI′, and J′ have a value whose fractional portion is equal to 0.5.Therefore, in the specific example shown in FIG. 20, positions denotedby solid circles such as (I−0.5, J−0.5) are regarded as positions (I′,J′).

As shown in FIG. 20, a spring model is defined for four true green lightintensities Lg(I−0.5, J−0.5), Lg(I−0.5, J+0.5), Lg(I+0.5, J−0.5), andLg(I+0.5, J+0.5) in the vicinity of the point A(x, y). Similarly, inaddition to the point A(x, y), a spring model using four springs isdefined for each point at which a pixel value Gobs(k, ig, jg) of firstto N-th captured images is observed. Thus, for the position (I′, J′), asshown in FIG. 21, the spring model is defined so as to represent therelationship between the true green light intensity Lg(I′, J′) at theposition (I′, J′) and pixel values Gobs(k, ig, jg) observed, forexample, at points A to E in the vicinity of the position (I′, J′).

That is, for the given position (I′, J′) of interest, all combinationsof integral values of k, ig, and jg are determined which satisfyI′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to the position (x, y) in thereference coordinate system obtained by converting, based on conversionparameters (θk, Tkx, Tky, Sk), the position (ig−0.5, jg−0.5). By way ofexample, let us assume that combinations of integers (k, ig, jg)identifying respective five pixel values Gobs(k, ig, jg) observed atpoints A to E are obtained for the position (I′, J′) as shown in FIG.21.

In this case, as shown in FIG. 22, a spring model is defined torepresent the relationship between the true green light intensity Lg(I′,J′) at the position (I′, J′) and each of the five pixel values Gobs(k,ig, jg) observed at points A to E. That is, for the given position (I′,J′) at which the true green light intensity Lg(I′, J′) is to bedetermined, the relationship between the true green light intensityLg(I′, J′) and each observed pixel value Gobs(k, ig, jg) is representedby the spring model.

FIG. 23 shows a method of estimating a true green light intensity Lg(I′,J′) at the position (I′, J′), based on the spring model associated withthe position (I′, J′) of interest. In FIG. 23, an axis represents thegreen light intensity (G signal).

In the spring model shown in FIG. 23, one end of each of five springs isconnected to an object V with a mass of 0, and the other end of eachspring is connected to one of five pixel values Gobs(k, ig, jg) atpoints A to E. That is, five pixel values Gobs(k, ig, jg) at respectivepoints A to E correspond to combinations of (k, ig, jg) that satisfyI′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to the position (x, y) in thereference coordinate system obtained by converting, based on conversionparameters (θk, Tkx, Tky, Sk), the position (ig−0.5, jg−0.5) for thegiven position (I′, J′) shown in FIG. 22.

The five springs have a natural length of 0. A spring whose end isconnected to a pixel value Gobs(k, ig, jg) observed at a point (x, y)has a spring constant equal to {√2−F((x, y), (I′, J′))}/(Mk×E), asdescribed earlier.

The five springs pull the object V toward the five pixel values Gobs(k,ig, jg), respectively, by forces proportional to the spring constants ofthe respective springs, and an equilibrium is achieved when the object Vis at a particular location. The light intensity obtained at thelocation at which the equilibrium is achieved is employed as an estimateof the true green light intensity Lg(I′, J′). The springs are in anequilibrium state when the sum of forces applied to the object V isequal to 0. That is, the equilibrium is achieved when the followingequation is satisfied. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Gobs}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (25)\end{matrix}$

Hereinafter, equation (25) will be referred to as a spring equationassociated with the green light intensity. For the given position (I′,J′), the sum Σ in equation (25) is taken for all combinations (k, ig,jg) that satisfy I′−1≦x<I′+1 and j′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), theposition (ig−0.5, jg−0.5). More specifically, in the example shown inFIG. 23, the sum is taken for five combinations of integers (k, ig, jg)corresponding to points A to E.

Equation (25) is in the form of a linear equation including Lg(I′, J′)as an unknown. By solving equation (25), the true green light intensityLg(I′, J′) at the position (I′, J′) can be determined.

For the true red light intensity Lr(I′, J′) and the true blue lightintensity Lb(I′, J′) at the position (I′, J′), linear equations (26) and(27) can be written in a similar manner to that in which equation (25)is written for the pixel that senses the green light component asdescribed above with reference to FIGS. 18 to 23. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Robs}\left( {k,{ir},{jr}} \right)} - {{Lr}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (26)\end{matrix}$

Hereinafter, equation (26) will be referred to as a spring equationassociated with the red light intensity. For the given position (I′,J′), the sum E in equation (26) is taken for all combinations (k, ir,jr) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), theposition (ir−0.5, jr−0.5). $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Bobs}\left( {k,{ib},{jb}} \right)} - {{Lb}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (27)\end{matrix}$

Hereinafter, equation (27) will be referred to as a spring equationassociated with the blue light intensity. For the given position (I′,J′), the sum E in equation (27) is taken for all combinations (k, ib,jb) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), theposition (ib−0.5, jb−0.5).

Thus, in this third embodiment, for the given position (I′, J′), thetrue green light intensity Lg(I′, J′), the true red light intensityLr(I′, J′), and the true blue light intensity Lb(I′, J′) are determinedbased on pixel values corresponding to combinations (k, ib, jb) thatsatisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to the position (x, y)in the reference coordinate system obtained by converting, based onconversion parameters (θk, Tkx, Tky, Sk), the position (i−0.5, j−0.5).

Now, referring to a flow chart shown in FIG. 24, a correct imageestimation process in step S4 shown in FIG. 2 according to a thirdembodiment of the present invention is described.

First, in step S71, the calculation circuit 24 selects a position (I′,J′) in the reference coordinate system (hereinafter, such a selectedposition will be referred to a position (I′, J′) of interest). Note thatthe position (I′, J′) of interest is at the center (i−0.5, j−0.5) of a(i-th, j-th) pixel of the first image taken as the reference image.

Thereafter, the process proceeds from step S71 to step S72. In step S72,the calculation circuit 24 determines all combinations of integers (k,ig, jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel that senses a greencomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS73.

In step S73, the calculation circuit 24 produces a spring equation inthe form (25) for green light intensity, using all combinations of (k,ig, jg) determined in step S72. Thereafter, the process proceeds to stepS74.

In step S74, the calculation circuit 24 determines all combinations ofintegers (k, ir, jr) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ir−0.5, jr−0.5) of the pixel that senses a redcomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS75.

In step S75, the calculation circuit 24 produces a spring equation inthe form (26) for red light intensity, using all combinations of (k, ir,jr) determined in step S74. Thereafter, the process proceeds to stepS76.

In step S76, the calculation circuit 24 determines all combinations ofintegers (k, ib, jb) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ib−0.5, jb−0.5) of the pixel that senses ablue component of the k-th image, for the position (I′, J′) of interestof each of first to N-th images. Thereafter, the process proceeds tostep S77.

In step S77, the calculation circuit 24 produces a spring equation inthe form (27) for blue light intensity, using all combinations of (k,ib, jb) determined in step S76. Thereafter, the process proceeds to stepS78.

In step S78, the calculation circuit 24 solves the spring equation (25)associated with the green light intensity determined in step S73, thespring equation (26) associated with the red light intensitydetermined-in step S75, and the spring equation (27) associated with theblue light intensity determined in step S77, which are all in the formof a linear equation, thereby determining the true green light intensityLg(I′, J′), the true red light intensity Lr(I′, J′), and the true bluelight intensity Lb(I′, J′) at the position (I′, J′) of interest.Thereafter, the process proceeds to step S79.

In step S79, the calculation circuit 24 determines whether theabove-described process is completed for all possible positions (I′,J′), that is, whether the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, J′), and the true blue light intensityLb(I′, J′) have been determined by taking the center position of each ofall pixels of the first image as the position (I′, J′) of interest.

If it is determined in step S79 that the process is not completed forall possible positions (I′, Jo) of interest, the process returns to stepS71 to repeat steps S71 to S79. That is, the calculation circuit 24selects, as a new position of interest, a position (I′, J′) that has notyet been selected as the position of infest, and determines the truegreen light intensity Lg(I′, J′), the true red light intensity Lr(I′,J′), and the true blue light intensity Lb(I′, J′) at the position (I′,J′) of interest.

On the other hand in the case in which it is determined in step S79 thatthe process is completed by taking all possible position (I′, J′) as theposition of interest, the process proceeds to step S80. In step S80, thecalculation circuit 24 estimates a correct image (signal) from the truegreen light intensity Lg(I′, J′), the true red light intensity Lr(I′,J′), and the true blue light intensity Lb(I′, J′) determined in stepS78. The resultant image is supplied as an output image to the D/Aconverter 9 or the codec 12, and the current process is exited. Forexample, for the (i-th, j-th) pixel, the calculation circuit 24estimates (produces) the correct image signal thereof from the truegreen light intensity Lg(i−0.5, j−0.5) determined as the green value (Gsignal) in step S78, the true red light intensity Lr(i−0.5, j−0.5)determined as the red value (R signal) in step S78, and the true bluelight intensity Lb(i−0.5, j−0.5) determined as the blue value (B signal)in step S78. The calculation circuit 24 estimates the output image byestimating the image signal for all pixels whose center is located atthe selected position (I′, J′).

In the third embodiment, as described above, data sensed by each pixelof the imaging device 4 is regarded as point-sampled data, therelationship between a pixel value observed at the center of each pixeland an ideal image signal of a sharp image including no blur due to acamera shake is represented by the spring model, and a sharp imagelikely to be a true image is determined based on the spring model.

Now, a correct image estimation process according to a fourth embodimentof the present invention is described. In this fourth embodiment, thethird embodiment described above with reference to FIG. 24 is partiallyimproved.

That is, in the third embodiment shown in FIG. 24, in step S72, thecalculation circuit 24 determines all combinations of integers (k, ig,jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel that senses a greencomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. In step S73, the calculation circuit 24produces a spring equation in the form (25) for green light intensity,using all combinations of integers (k, ig, jg) determined in step S72.Similarly, for the pixels that sense the red or blue component,combinations of integers (k, ig, jg) are determined for the position(I′, J′) of interest, and the spring equation in the form of (26) or(27) is produced using all determined combinations of integers (k, ig,jg).

There is a possibility that data of a particular pixel of a particularimage of the first to N-th images is unreliable depending on the statein which the images are captured. In the digital camera 1 shown in FIG.1, N images are taken and stored in the frame memory 22. Therefore, evenif data with low reliability is discarded, the amount of remaining datacan be sufficiently large.

In view of the above, in a fourth embodiment, only data with highreliability is used (and data with low reliability is discarded) toproduce an image with better sharpness. More specifically, in the fourthembodiment, as many combinations of integers (k, ig, jg) as apredetermined number L are extracted as data with high reliability fromall (k, ig, jg) determined for a position (I′, J′) of interest. Thenumber L may be fixed to a predetermined value such as 8 or may be setto a variable value specified by a user. The calculation circuit 24calculates the mean value of pixel values Gobs(k, ig, jg) of allcombinations of integers (k, ig, jg) and selects L (=8) combinations ofintegers (k, ig, jg) whose pixel value Gobs(k, ig, jg) is closest to themean value. The selected L combinations are used to determine the truegreen light intensity Lg(I′, J′) at the position (I′, J′).

Thus, in this fourth embodiment, the spring equations (25), (26), and(27) associated with the green, red and blue light intensities used inthe third embodiment are respectively rewritten into equations (28),(29) and (30), as shown below. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Gobs}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (28)\end{matrix}$where Σ represents the sum taken for L combinations of integers (k, ig,jg) whose pixel value Gobs(k, ig, jg) is closest to the mean value ofpixel values Gobs(k, ig, jg) of all combinations of integers (k, ig, jg)determined for the position (I′, J′) of interest. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Robs}\left( {k,{ir},{jr}} \right)} - {{Lr}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (29)\end{matrix}$where Σ represents the sum taken for L combinations of integers (k, ir,jr) whose pixel value Robs(k, ir, jr) is closest to the mean value ofpixel values Robs(k, ir, jr) of all combinations of integers (k, ir, jr)determined for the position (I′, J′) of interest. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Bobs}\left( {k,{ib},{jb}} \right)} - {{Lb}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (30)\end{matrix}$where Σ represents the sum taken for L combinations of integers (k, ib,jb) whose pixel value Bobs(k, ib, jb) is closest to the mean value ofpixel values Bobs(k, ib, jb) of all combinations of integers (k, ib, jb)determined for the position (I′, J′) of interest.

Now, referring to a flow chart shown in FIG. 25, a correct imageestimation process in step S4 shown in FIG. 2 according to a fourthembodiment of the present invention is described.

First, in step S91, the calculation circuit 24 selects a position (I′,J′) in the reference coordinate system (hereinafter, such a selectedposition will be referred to a position (I′, J′) of interest). Note thatthe position (I′, J′) of interest is at the center (i−0.5, j−0.5) of a(i-th, j-th) pixel of the first image taken as the reference image.

Thereafter, the process proceeds from step S91 to step S92. In step S92,the calculation circuit 24 determines all combinations of integers (k,ig, jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel that senses a greencomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS93.

In step S93, the calculation circuit 24 calculates the mean value ofpixel values Gobs(k, ig, jg) for all combinations of integers (k, ig,jg) determined in step S92, and selects L combinations of integers (k,ig, jg) whose pixel value Gobs(k, ig, jg) is closest to the mean value.Thereafter, the process proceeds to step S94. That is, in step S93described above, pixel values Gobs(k, ig, jg) that are not close to themean value taken for all combinations of integers (k, ig, jg) determinedin step S92 are regarded as data with low reliability and they arediscarded. In a case in which the total number of combinations ofintegers (k, ig, jg) determined in step S92 is less than L, allcombinations of integers (k, ig, jg) are selected (without discardingany data).

In step S94, the calculation circuit 24 produces a spring equation inthe form (28) for green light intensity, using L combinations of (k, ig,jg) selected in step S93. Thereafter, the process proceeds to step S95.

In step S95, the calculation circuit 24 determines all combinations ofintegers (k, ir, jr) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ir−0.5, jr−0.5) of the pixel that senses a redcomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS96.

In step S96, the calculation circuit 24 calculates the mean value ofpixel values Robs(k, ir, jr) for all combinations of integers (k, ir,jr) determined in step S96, and selects L combinations of integers (k,ir, jr) whose pixel value Robs(k, ir, jr) is closest to the mean value.Thereafter, the process proceeds to step S97. That is, in step S96described above, pixel values Robs(k, ir, jr) that are not close to themean value taken for all combinations of integers (k, ir, jr) determinedin step S95 are regarded as data with low reliability and they arediscarded. In a case in which the total number of combinations ofintegers (k, ir, jr) determined in step S95 is less than L, allcombinations of integers (k, ir, jr) are selected (without discardingany data).

In step S97, the calculation circuit 24 produces a spring equation inthe form (29) for red light intensity, using L combinations of (k, ir,jr) selected in step S96. Thereafter, the process proceeds to step S98.

In step S98, the calculation circuit 24 determines all combinations ofintegers (k, ib, jb) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ib−0.5, jb−0.5) of the pixel that senses ablue component of the k-th image, for the position (I′, J′) of interestof each of first to N-th images. Thereafter, the process proceeds tostep S99.

In step S99, the calculation circuit 24 calculates the mean value ofpixel values Bobs(k, ib, jb) for all combinations of integers (k, ib,jb) determined in step S98, and selects L combinations of integers (k,ib, jb) whose pixel value Bobs(k, ib, jb) is closest to the mean value.Thereafter, the process proceeds to step S100. That is, in step S99described above, pixel values Bobs(k, ib, jb) that are not close to themean value taken for all combinations of integers (k, ib, jb) determinedin step S98 are regarded as data with low reliability and they arediscarded. In a case in which the total number of combinations ofintegers (k, ib, jb) determined in step S98 is less than L, allcombinations of integers (k, ib, jb) are selected (without discardingany data).

In step S100, the calculation circuit 24 produces a spring equation inthe form (30) for blue light intensity, using L combinations of (k, ib,jb) selected in step S99. Thereafter, the process proceeds to step S1ol.

In step S101, the calculation circuit 24 solves the spring equation (28)associated with the green light intensity determined in step S94, thespring equation (29) associated with the red light intensity determinedin step S97, and the spring equation (30) associated with the blue lightintensity determined in step S100, which are all in the form of a linearequation, thereby determining the true green light intensity Lg(I′, J′),the true red light intensity Lr(I′, J′), and the true blue lightintensity Lb(I′, J′) at the position (I′, J′) of interest. Thereafter,the process proceeds to step S102.

In step S102, the calculation circuit 24 determines whether theabove-described process is completed for all possible positions (I′,J′), that is, whether the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, J′), and the true blue light intensityLb(I′, J′) have been determined by taking the center position of each ofall pixels of the first image as the position (I′, J′) of interest.

If it is determined in step S102 that the process is not completed forall possible positions (I′, J′) of interest, the process returns to stepS91 to repeat steps S91 to S102. That is, the calculation circuit 24selects, as a new position of interest, a position (I′, J′) that has notyet been selected as the position of infest, and determines the truegreen light intensity Lg(I′, J′), the true red light intensity Lr(I′,J′), and the true blue light intensity Lb(I′, J′) at the position (I′,J′) of interest.

On the other hand in the case in which it is determined in step S102that the process is completed by taking all possible position (I′, J′)as the position of interest, the process proceeds to step S103. In stepS103, the calculation circuit 24 estimates a correct image (signal) fromthe true green light intensity Lg(I′, J′), the true red light intensityLr(I′, J′), and the true blue light intensity Lb(I′, J′) determined instep S101. The resultant image is supplied as an output image to the D/Aconverter 9 or the codec 12, and the current process is exited. Forexample, for the (i-th, j-th) pixel, the calculation circuit 24estimates a correct image signal thereof from the true green lightintensity Lg(i−0.5, j−0.5) determined as the green value (G signal) instep S101, the true red light intensity Lr(i−0.5, j−0.5) determined asthe red value (R signal) in step S101, and the true blue light intensityLb(i−0.5, j−0.5) determined as the blue value (B signal) in step S101.The calculation circuit 24 estimates a correct image by performing theabove-described process for all pixels whose center is located at theselected position (I′, J′).

In the fourth embodiment, as described above, pixel values close to themean value are employed as reliable data, and the spring model isemployed to only the employed reliable data. This makes it possible toobtain a sharper image more likely to be a correct image than can beobtained by the third embodiment described above.

Now, a correct image estimation process according to a fifth embodimentof the present invention is described. In this fifth embodiment, as inthe fourth embodiment, as many combinations of integers (k, ig, jg) as apredetermined number L are extracted as reliable data from allcombinations of integers (k, ig, jg) determined for a position (I′, J′)of interest. The true green light intensity Lg(I′, J′), the true redlight intensity Lr(I′, J′), and the true blue light intensity Lb(I′, J′)are determined by solving spring equations associated with green, red,and blue light intensities each using L reliable data.

In the fourth embodiment described above, for example, in thedetermination of the true green light intensity Lg(I′, J′), the meanvalue of pixel values Gobs(k, ig, jg) of all combinations of integers(k, ig, jg) is determined by the calculation circuit 24, and Lcombinations of integers (k, ig, jg) whose pixel value Gobs(k, ig, jg)is closest to the mean value are selected.

On the other hand, in the fifth embodiment, for a given position (I′,J′) of interest, the calculation circuit 24 extracts reliable data byselecting L combinations of integers (k, ig, jg) that are shortest indistance from the position of interest (I′, J′) to the position (x, y)in the reference coordinate system obtained by converting, based onconversion parameters (θk, Tkx, Tky, Sk), the center position (ig−0.5,jg−0.5) of the pixel which senses the green component of the k-th image.This is because, as described earlier with reference to FIG. 19, theapproximation error that can occur when the true green light intensityLg(I′, J′) at the position (I′, J′) of interest is approximated by thepixel value Gobs(k, ig, jg) observed at the position (x, y) increaseswith the distance between the position (I′, J′) of interest and theposition (x, y). In other words, pixel values Gobs(k, ig, jg) observedat positions (x, y) distant from the position (I′, J′) of interest arelow in reliability. To determined the distance between two points (x1,y1) and (x2, y2), for example, the function F((x1, y1), (x2, y2))described earlier with reference to FIG. 19 can be used.

In the fifth embodiment, spring equations (31), (32), and (33)associated with green, red, and blue light intensities shown below areused instead of the spring equations (25), (26), and (27) used in thethird embodiment. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Gobs}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (31)\end{matrix}$where Σ denotes the sum taken for L combinations of integers (k, ig, jg)that are shortest in distance from the position (I′, J′) of interest tothe position (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel which senses the greencomponent of the k-th image. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Robs}\left( {k,{ir},{jr}} \right)} - {{Lr}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (32)\end{matrix}$where Σ denotes the sum taken for L combinations of integers (k, ir, jr)that are shortest in distance from the position (I′, J′) of interest tothe position (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ir−0.5, jr−0.5) of the pixel which senses the redcomponent of the k-th image. $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Bobs}\left( {k,{ib},{jb}} \right)} - {{Lb}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (33)\end{matrix}$where Σ denotes the sum taken for L combinations of integers (k, ib, jb)that are shortest in distance from the position (I′, J′) of interest tothe position (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ib−0.5, jb−0.5) of the pixel which senses the bluecomponent of the k-th image.

Now, referring to a flow chart shown in FIG. 26, a correct imageestimation process in step S4 shown in FIG. 2 according to a fifthembodiment of the present invention is described.

First, in step S121, the calculation circuit 24 selects a position (I′,J′) in the reference coordinate system (hereinafter, such a selectedposition will be referred to a position (I′, J′) of interest). Note thatthe position (I′, J′) of interest is at the center (i−0.5, j−0.5) of a(i-th, j-th) pixel of the first image taken as the reference image.

Thereafter, the process proceeds from step S121 to step S122. In stepS122, the calculation circuit 24 determines all combinations of integers(k, ig, jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel that senses a greencomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS123.

In step S123, from all combinations of integers (k, ig, jg) determinedin step S122, the calculation circuit 24 selects L combinations ofintegers (k, ig, jg) that are shortest in distance from the position ofinterest (I′, J′) to the position (x, y) in the reference coordinatesystem obtained by converting, based on conversion parameters (θk, Tkx,Tky, Sk), the center position (ig−0.5, jg−0.5) of the pixel which sensesthe green component of the k-th image. Thereafter, the process proceedsto step S124. That is, in step S123 described above, of all combinationsof integers (k, ig, jg) determined in step S122, those that are long indistance between the position (x, y) and the position (I′, J′) ofinterest are regarded as data with low reliability and they arediscarded. In a case in which the total number of combinations ofintegers (k, ig, jg) determined in step S122 is less than L, allcombinations of integers (k, ig, jg) are selected (without discardingany data).

In step S124, the calculation circuit 24 produces a spring equation inthe form (31) for green light intensity, using L combinations of (k, ig,jg) selected in step S123. Thereafter, the process proceeds to stepS125.

In step S125, the calculation circuit 24 determines all combinations ofintegers (k, ir, jr) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ir−0.5, jr−0.5) of the pixel that senses a redcomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS126.

In step S126, from all combinations of integers (k, ir, jr) determinedin step S125, the calculation circuit 24 selects L combinations ofintegers (k, ir, jr) that are shortest in distance from the position ofinterest (I′, J′) to the position (x, y) in the reference coordinatesystem obtained by converting, based on conversion parameters (θk, Tkx,Tky, Sk), the center position (ir−0.5, jr−0.5) of the pixel which sensesthe green component of the k-th image. Thereafter, the process proceedsto step S127. That is, in step S126 described above, of all combinationsof integers (k, ir, jr) determined in step S125, those that are long indistance between the position (x, y) and the position (I′, J′) ofinterest are regarded as data with low reliability and they arediscarded. In a case in which the total number of combinations ofintegers (k, ir, jr) determined in step S125 is less than L, allcombinations of integers (k, ir, jr) are selected (without discardingany data).

In step S127, the calculation circuit 24 produces a spring equation inthe form (32) for red light intensity, using L combinations of (k, ir,jr) selected in step S126. Thereafter, the process proceeds to stepS128.

In step S128, the calculation circuit 24 determines all combinations ofintegers (k, ib, jb) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ib−0.5, jb−0.5) of the pixel that senses ablue component of the k-th image, for the position (I′, J′) of interestof each of first to N-th images. Thereafter, the process proceeds tostep S129.

In step S129, from all combinations of integers (k, ib, jb) determinedin step S128, the calculation circuit 24 selects L combinations ofintegers (k, ib, jb) that are shortest in distance from the position ofinterest (I′, J′) to the position (x, y) in the reference coordinatesystem obtained by converting, based on conversion parameters (θk, Tkx,Tky, Sk), the center position (ib−0.5, jb−0.5) of the pixel which sensesthe green component of the k-th image. Thereafter, the process proceedsto step S130. That is, in step S129 described above, of all combinationsof integers (k, ib, jb) determined in step S128, those that are long indistance between the position (x, y) and the position (I′, J′) ofinterest are regarded as data with low reliability and they arediscarded. In a case in which the total number of combinations ofintegers (k, ib, jb) determined in step S128 is less than L, allcombinations of integers (k, ib, jb) are selected (without discardingany data).

In step S130, the calculation circuit 24 produces a spring equation inthe form (33) for blue light intensity, using L combinations of (k, ib,jb) selected in step S129. Thereafter, the process proceeds to stepS131.

In step S131, the calculation circuit 24 solves the spring equation (31)associated with the green light intensity determined in step S124, thespring equation (32) associated with the red light intensity determinedin step S127, and the spring equation (33) associated with the bluelight intensity determined in step S130, which are all in the form of alinear equation, thereby determining the true green light intensityLg(I′, J′), the true red light intensity Lr(I′, J′), and the true bluelight intensity Lb(I′, J′) at the position (I′, J′) of interest.Thereafter, the process proceeds to step S132.

In step S132, the calculation circuit 24 determines whether theabove-described process is completed for all possible positions (I′,J′), that is, whether the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, J′), and the true blue light intensityLb(I′, J′) have been determined by taking the center position of each ofall pixels of the first image as the position (I′, J′) of interest.

If it is determined in step S132 that the process is not completed forall possible positions (I′, J′) of interest, the process returns to stepS121 to repeat steps S121 to S132. That is, the calculation circuit 24selects, as a new position of interest, a position (I′, J′) that has notyet been selected as the position of infest, and determines the truegreen light intensity Lg(I′, J′), the true red light intensity Lr(I′,J′), and the true blue light intensity Lb(I′, J′) at the position (I′,J′) of interest.

On the other hand in the case in which it is determined in step S132that the process is completed by taking all possible position (I′, J′)as the position of interest, the process proceeds to step S133. In stepS133, the calculation circuit 24 estimates a correct image (signal) fromthe true green light intensity Lg(I′, J′), the true red light intensityLr(I′, J′), and the true blue light intensity Lb(I′, J′) determined instep S131. The resultant image is supplied as an output image to the D/Aconverter 9 or the codec 12, and the current process is exited. Forexample, for the (i-th, j-th) pixel, the calculation circuit 24estimates the correct image signal thereof from the true green lightintensity Lg(i−0.5, j−0.5) determined as the green value (G signal) instep S131, the true red light intensity Lr(i−0.5, j−0.5) determined asthe red value (R signal) in step S131, and the true blue light intensityLb(i−0.5, j−0.5) determined as the blue value (B signal) in step S131.The calculation circuit 24 estimates the output image by estimating theimage signal for all pixels whose center is located at the selectedposition (I′, J′).

In the fifth embodiment, as described above, pixel values observed atpositions close to the position (I′, J′) of interest are selected asreliable data, and the spring model is applied only to the selecteddata. This makes it possible to obtain a more ideal image with bettersharpness than can be obtained by the third embodiment described above.

Now, a correct image estimation process according to a sixth embodimentof the present invention is described.

In this sixth embodiment, the third embodiment described above isfurther improved. That is, in this sixth embodiment, edge parts of animage are detected, and spring equations in improved forms based onspring equations (25), (26), and (27) associated with green, red andblue light intensities are applied to pixel values in the detected edgeparts.

Of green, red, and blue components, a reference coordinate systemassociated with the green component (G signal) is shown in FIG. 27.

In the example, shown in FIG. 27, pixel values with bright green areobserved in an area to the right of a boundary (edge) 51, and pixelvalues with dark green are observed in an area to the left of theboundary (edge) 51. Each pixel value of green is represented in, forexample, 8 bits. The bright green pixel value is equal to, for example,240, and the dark green pixel value is equal to, for example, 16.

In FIG. 27, open circles 52-1 to 52-10, 53-1 to 53-9, and 54 indicatecenter positions (ig−0.5, jg−0.5) of (ig-th, jg-th) pixels of the k-thimage converted, based on conversion parameters (θk, Tkx, Tky, Sk), torepresentation in the reference coordinate system. At those positions,green pixel values Gobs(k, ig, jg) are observed. Note that (k, ig, jg)is different among open circles 52-1 to 52-10, 53-1 to 53-9, and 54.

Each solid circle at an intersection of a lattice represented by dottedlines in FIG. 27 indicates a position (I′, J′) of green light intensityLg(I′, J′) to be determined by the calculation circuit 24. As describedearlier, the position (I′, J′) denotes the center position (i−0.5,j−0.5) of the (i-th, j-th) pixel of the first image employed as thereference image. The distance between adjacent Lg(I′, J′) is equal to 1in both X and Y directions, and thus the pixel size is equal to 1 inboth X and Y directions.

At positions denoted by open circles 52-1 to 52-8 in a region 61 shownin FIG. 27, bright green pixel values Gobs(k, ig, jg) equal to about 240are observed.

At positions denoted by open circles 53-1 to 53-7 in a region 62 shownin FIG. 27, dark green pixel values Gobs(k, ig, jg) equal to about 16are observed.

In FIG. 27, the position indicated by the open circle 54 is on theboundary 51. At such a position, because of the characteristic of theon-chip lens, the pixel receives equally both bright green light with alevel of 240 and dark green light with a level of 16, and thus the pixelvalue Gobs(k, ig, jg) observed at the position denoted by the opencircle 54 is equal to the mean value of 240 and 16, that is,128(=(240+16)/2).

The pixel values Gobs(k, ig, jg) observed at positions denoted by opencircles 52-1 to 52-8, 53-1 to 53-7 and 54 generally include an errorcomponent, and thus, from a strict point of view, pixel values are, forexample, about 240, about 16, about 128, and so on. However, for thepurpose of simplicity, error components are ignored and pixel values aresimply expressed as 240, 16, 128, and so on.

For example, when the position denoted by the solid circle 71 in FIG. 27is selected as the position (I′, J′) of interest, if the thirdembodiment described above is used, the true green light intensityLg(I′, J′) at the position (I′, J′) of interest denoted by the solidcircle 71 is determined as follows.

That is, when the position denoted by the solid circle 71 is selected asthe position (I′, J′) of interest, pixel values Gobs(k, ig, jg) observedin the vicinity of the position (I′, J′) of interest, such as thoseobserved at positions denoted by open circles 52-3, 52-6, 52-10, and 54,are employed as Gobs(k, ig, jg) in equation (25). In this case, thegreen light intensity Lg(I′, J′) obtained by solving equation (25)becomes lower than 240 because the true green light intensity Lg(I′, J′)is pulled (affected) by the pixel value of 128 observed at the positiondenoted by the open circle 54. However, because the position denoted bythe solid circle 71 is located in an area to the right of the boundary51 where the green pixel value observed is as high as 240, the reallytrue green light intensity Lg(I′, J′) is 240. Thus, it is desirable thatthe green light intensity Lg(I′, J′) obtained by solving equation (25)be equal to 240.

When the position denoted by the solid circle 72 is selected as theposition (I′, J′) of interest, pixel values Gobs(k, ig, jg) observed inthe vicinity of the position (I′, J′) of interest, such as thoseobserved at positions denoted by open circles 53-2, 53-3, 53-5, 53-9 and54 are employed as Gobs(k, ig, jg) in equation (25). In this case, thetrue green light intensity Lg(I′, J′) obtained by solving equation (25)becomes greater than 16 because the true green light intensity Lg(I′,J′) is pulled (affected) by the pixel value of 128 observed at theposition denoted by the open circle 54. However, because the positiondenoted by the solid circle 72 is located in an area to the left of theboundary 51 where the green pixel value observed is as low as 16, thereally true green light intensity Lg(I′, J′) is 16. Thus, it isdesirable that the green light intensity Lg(I′, J′) obtained by solvingequation (25) be equal to 16.

The above discussion can also be applied to locations denoted by solidcircles 73 and 74. As discussed above, in images produced by the correctimage estimation process according to the third embodiment, in edgeparts such as that in which the boundary 51 shown in FIG. 27 exists,there is a possibility that an edge is blurred (that is, differencesamong pixel values of adjacent pixels become small).

In the sixth embodiment, to avoid the above problem, an edge partincluding the boundary 51 shown in FIG. 27 is detected, and the specialprocess is performed on pixel values Gobs(k, ig, jg) in the detectededge part, that is, pixel values Gobs(k, ig, jg) substituted into thespring equation (25) associated with the green light intensity arechanged.

The method according to the sixth embodiment is described in furtherdetail below for the case in which the green light intensity (G signal)is determined with reference to FIG. 27.

First, the calculation circuit 24 detects an edge part based on allpixel values Gobs(k, ig, jg) at positions converted into the referencecoordinate system. That is, the calculation circuit 24 determineswhether there is an edge extending in a vertical direction, in ahorizontal direction, in a direction from upper left to lower right, orin a direction from lower right to lower left.

Referring to FIG. 28, a method of determining whether there is avertical edge is described below.

FIG. 28 shows a reference coordinate system. In FIG. 28, open circles81-1 to 81-10, 82-1 to 82-9, and 83 indicate center positions (ig−0.5,jg−0.5) of (ig-th, jg-th) pixels of the k-th image converted, based onconversion parameters (θk, Tkx, Tky, Sk), to representation in thereference coordinate system, and pixel values Gobs(k, ig, jg) areobserved at those positions. Note that (k, ig, jg) is different amongopen circles 81-1 to 81-10, 82-1 to 82-9, and 83.

Each solid circle at an intersection of a lattice represented by dottedlines in FIG. 28 indicates a position (I′, J′) of green light intensityLg(I′, J′) to be determined by the calculation circuit 24. As describedearlier, the position (I′, J′) denotes the center position (i−0.5,j−0.5) of the (i-th, j-th) pixel of the first image employed as thereference image. As in the example shown in FIG. 27, the distancebetween adjacent Lg(I′, J′) is equal to 1 in both X and Y directions,and thus the pixel size is equal to 1 in both X and Y directions.

For example, the calculation circuit 24 selects a position denoted by anopen circle 83 shown in FIG. 28 as a position of interest, anddetermines whether there is a vertical edge such as an edge 94 shown inFIG. 28. Herein, let us assume that the position (x, y) denoted by theopen circle 83 satisfy inequality I−0.5≦x<I+0.5and J−0.2≦y<J+0.5. In theexample shown in FIG. 28, the position (x, y) denoted by the open circle83 is located in a rectangular area All whose four corners are locatedat (I +0.5, J−0.5) denoted by a solid circle 84, (I−0.5, J−0.5) denotedby a solid circle 85, (I+0.5, J+0..5) denoted by a solid circle 86, and(I−0.5, J+0.5) denoted by a solid circle 87. I and J are integers as inthe previous embodiments.

The calculation circuit 24 defines nine regions each having a size of 1equal to the size of one pixel such that a region A11 including an opencircle 83 of interest is located at the center of these nine regions.These nine regions are: a region A00 defined by I−1.5≦x<I−0.5 andJ−1.5≦y<J−0.5; a region A01 defined by I−1.5≦x<I−0.5 and J−0.5≦y<J+0.5;a region A02 defined by I−1.5≦x<I−0.5 and J+0.5≦y<J+1.5; a region Alodefined by I−0.5≦x<I+0.5and J−1.5≦y<J−0.5; the region A11 defined byI−0.5≦x<I+0.5and J−0.5≦y<J+0.5; a region A12 defined by I−0.5≦x<I+0.5and J+0.5≦y<J+1.5; a region A20 defined by I+0.2≦x<I+0.5andJ−1.5≦y<J−0.5; a region A21 defined by I+0.5≦x<I+1.5and J−0.5≦y<J+0.5;and a region A22 defined by I+0.5≦x<I+0.5and J+0.5≦y<J+1.5. The currentposition of interest denoted by the open circle 83 is in the region A11.

The total area including the nine regions A00 to A22 defined for theposition of interest denoted by the open circle 83 is referred to as anedge judgment region.

To determine whether there is a horizontal edge such as the edge 94, thecalculation circuit 24 calculates the mean value and the variance ofpixel values Gobs(k, ig, jg) at locations denoted by open circles in aleft region 91 including a region A00, a region A01, and a region A02 inthe edge judgment region (hereinafter, this region 91 will be referredto simply as the left region 91), the mean value and the variance ofpixel values Gobs(k, ig, jg) at locations denoted by open circles in acentral region 92 including a region A10, a region A11, and a region A12in the edge judgment region (hereinafter, this region 92 will bereferred to simply as the central region 92), and the mean value and thevariance of pixel values Gobs(k, ig, jg) at locations denoted by opencircles in a right region 93 including a region A20, a region A21, and aregion A22 in the edge judgment region (hereinafter, this region 93 willbe referred to simply as the right region 93).

That is, the calculation circuit 24 determines the mean value EG0 ofpixel values Gobs(k, ig, jg) at positions denoted by open circles 82-1to 82-7 in a left-hand region 91 and also determines the variance SGO ofthose pixel values Gobs(k, ig, jg). The calculation circuit 24 alsodetermines the mean value EG0″ of pixel values Gobs(k, ig, jg) atpositions denoted by open circles 81-9, 81-10, 82-8, 82-9, and 83 in acentral region 92. Furthermore, the calculation circuit 24 determinesthe mean value EG0′ of pixel values Gobs(k, ig, jg) at positions denotedby open circles 81-1 to 81-8 in a right-hand region 93 and alsodetermines the variance SGO′ of those pixel values Gobs(k, ig, jg).

By way of example, if there is a vertical edge 94 close to the positionof interest denoted by the open circle 83, the following inequality (a)holds for the mean value EG0 of the left region 91, the mean value EG0″of the central region 92, and the mean value EG0′ of the right region93, and thus the calculation circuit 24 determines that there is avertical edge when inequality (a) is satisfied.EG0<EG0″<EG0′ or EG0′<EG0″<EG0  (a)

In practice, inequality (a) can be satisfied for an area that is notclose to the edge 94, if there is a variation in data due to, forexample, a fine pattern of a subject. To avoid the above problem and tomake a more reliable determination, the calculation circuit 24determines whether there is a vertical edge 94 by determining whetherthe following inequality including variances SG0 and SG0′ as additionalterms is satisfied, that is, the calculation circuit 24 determines thatthere is a vertical edge when inequality (b) shown below is satisfied.EG0+SG0<EG0″<EG0′−SG0′orEG0′+SG0′<EG0″<EG0−SG0  (b)

Use of inequality (b) makes it possible to prevent an area in whichpixel values have variations due to a fine pattern or the like and thusin which variances SG0 and SG0′are large from being incorrectlyidentified as an area in which there is an edge.

A method of determining whether there is a horizontal edge is describedbelow with reference to FIG. 29. In FIG. 29, similar parts to those inFIG. 28 are denoted by similar reference numerals, and a duplicateddescription thereof is not given herein.

To determine whether there is a horizontal edge 104, the calculationcircuit 24 determines the mean value and the variance of pixel valuesGobs(k, ig, jg) at positions denoted by open circles in an upperedge-judgment region 101 extending in a horizontal direction andincluding a region A00, a region A10, and a region A20 (hereinafter, theregion 101 will be referred to simply as the upper region 101), the meanvalue and the variance of pixel values Gobs(k, ig, jg) at positionsdenoted by open circles in a central edge-judgment region 102 extendingin a horizontal direction and including a region A01, a region A11, anda region A21 (hereinafter, the region 102 will be referred to simply asthe central region 102), and the mean value and the variance of pixelvalues Gobs(k, ig, jg) at positions denoted by open circles in a loweredge-judgment region 103 extending in a horizontal direction andincluding a region A02, a region A12, and a region A22 (hereinafter, theregion 103 will be referred to simply as the lower region 103).

That is, the calculation circuit 24 determines the mean value EG1 ofpixel values Gobs(k, ig, jg) at positions denoted by open circles 81-1to 81-3, 81-10, 82-1, 82-2, and 82-9 in an upper region 101 and alsodetermines the variance SG1 of those pixel values Gobs(k, ig, jg). Thecalculation circuit 24 also determines the mean value EG1″ of pixelvalues Gobs(k, ig, jg) at positions denoted by open circles 81-4 to81-6, 82-3 to 82-5, and 83 in a central region 102. Furthermore, thecalculation circuit 24 determines the mean value EG1′ of pixel valuesGobs(k, ig, jg) at positions denoted by open circles 81-7 to 81-9 and82-6 to 82-8 in a lower region 103 and also determines the variance SG1′of those pixel values Gobs(k, ig, jg).

By way of example, if there is a horizontal edge 104 close to theposition of interest denoted by the open circle 83, the followinginequality (c) holds for the mean value EG1 of the upper region 101, themean value EGl″ of the central region 102, and the mean value EG1′ ofthe lower region 103 taking into account variations in data due to afine pattern of a subject, and thus the calculation circuit 24determines that there is a horizontal edge when inequality (c) issatisfied.EG1+SG1<EG1″<EG1′−SG1′orEG1′+SG1′<EG1″<EG1−SG1  (c)

Use of inequality (c) makes it possible to prevent an area in whichpixel values have variations due to a fine pattern or the like and thusin which variances SG1 and SG1′ are large from being incorrectlyidentified as an area in which there is an edge. Note that adetermination as to whether there is a horizontal edge may be made basedon inequality similar to (a) including only the mean values withouttaking into account variances.

A method of determining whether there is an edge extending from upperleft to lower right is described below with reference to FIG. 30. InFIG. 30, similar parts to those in FIG. 28 are denoted by similarreference numerals, and a duplicated description thereof is not givenherein.

To determine whether there is an edge 114 extending from upper left tolower right, the calculation circuit 24 determines the mean value andthe variance of pixel values Gobs(k, ig, jg) at positions denoted byopen circles in a lower-left edge-judgment region 111 including a regionA01, a region A02, and a region A12 (hereinafter, the region 111 will bereferred to simply as the lower left region 111), the mean value and thevariance of pixel values Gobs(k, ig, jg) at positions denoted by opencircles in a central edge-judgment region 112 including a region A00, aregion A11, and a region A22 (hereinafter, the region 112 will bereferred to simply as the central region 112), and the mean value andthe variance of pixel values Gobs(k, ig, jg) at positions denoted byopen circles in an upper-left edge-judgment region 113 including aregion A10, a region A20, and a region A21 (hereinafter, the region 113will be referred to simply as the upper left region 113).

That is, the calculation circuit 24 determines the mean value EG2 ofpixel values Gobs(k, ig, jg) at positions denoted by open circles 81-9and 82-3 to 82-8 in a lower-left region 111 and also determines thevariance SG2 of those pixel values Gobs(k, ig, jg). The calculationcircuit 24 also determines the mean value EG2′ of pixel values Gobs(k,ig, jg) at positions denoted by open circles 81-7, 81-8, 82-1, 82-2, and83 in a central region 112. Furthermore, the calculation circuit 24determines the mean value EG2′ of pixel values Gobs(k, ig, jg) atpositions denoted by open circles 81-1 to 81-6, 81-10, and 82-9 in anupper right region 113 and also determines the variance SG2′ of thosepixel values Gobs(k, ig, jg).

By way of example, if there is an edge 114 extending from upper left tolower right at a location close to the position of interest denoted bythe open circle 83, the following inequality (d) holds for the meanvalue EG2 of the left lower 111, the mean value EG2′ of the centralregion 112, and the mean value EG2′ of the upper right region 113,taking into account variations in data due to a fine pattern or the likeof a subject, and thus the calculation circuit 24 determines that thereis an edge extending from upper left to lower right when inequality (d)is satisfied.EG2+SG2<EG2″<EG2′−SG2′orEG2′+SG2′<EG2″<EG2−SG2  (d)

Use of inequality (d) makes it possible to prevent an area in whichpixel values have variations due to a fine pattern or the like and thusin which variances SG2 and SG2′ are large from being incorrectlyidentified as an area in which there is an edge. Note that adetermination as to whether there is an edge extending from upper leftto lower right may be made based on inequality similar to (a) includingonly the mean values without taking into account variances.

A method of determining whether there is an edge extending from upperright to lower left is described below with reference to FIG. 31. InFIG. 31, similar parts to those in FIG. 28 are denoted by similarreference numerals, and a duplicated description thereof is not givenherein.

To determine whether there is an edge 124 extending from upper right tolower left, the calculation circuit 24 determines the mean value and thevariance of pixel values Gobs(k, ig, jg) at positions denoted by opencircles in an upper-left edge-judgment region 121 including a regionA00, a region A01, and a region A10 (hereinafter, the region 121 will bereferred to simply as the upper left region 121), the mean value and thevariance of pixel values Gobs(k, ig, jg) at positions denoted by opencircles in a central edge-judgment region 122 including a region A02, aregion A11, and a region A20 (hereinafter, the region 122 will bereferred to simply as the central region 122), and the mean value andthe variance of pixel values Gobs(k, ig, jg) at positions denoted byopen circles in a lower-right edge-judgment region 123 including aregion A12, a region A21, and a region A22 (hereinafter, the region 123will be referred to simply as the lower right region 123).

That is, the calculation circuit 24 determines the mean value EG3 ofpixel values Gobs(k, ig, jg) at positions denoted by open circles 81-10,82-1 to 82-5, and 82-9 in an upper left 121 and also determines thevariance SG3 of those pixel values Gobs(k, ig, jg). The calculationcircuit 24 also determines the mean value EG3′ of pixel values Gobs(k,ig, jg) at positions denoted by open circles 81-1 to 81-3, 82-6, 82-7,and 83 in a central region 122. Furthermore, the calculation circuit 24determines the mean value EG3′ of pixel values Gobs(k, ig, jg) atpositions denoted by open circles 81-4 to 81-9 and 82-8 in a lower rightregion 123 and also determines the variance SG3′ of those pixel valuesGobs(k, ig, jg).

By way of example, if an edge 124 extending from upper left to lowerright is located close to the position of interest denoted by the opencircle 83, the following inequality (e) holds for the mean value EG3 ofthe upper left 121, the mean value EG3′ of the central region 122, andthe mean value EG3′ of the lower right 123 taking into accountvariations in data due to a fine pattern of a subject, and thus thecalculation circuit 24 determines that there is an edge extending fromupper left to lower right when inequality (e) is satisfied.EG3+SG3<EG3″<EG3′−SG3′orEG3′+SG3′<EG3″<EG3−SG3′  (e)

Use of inequality (e) makes it possible to prevent an area in whichpixel values have variations due to a fine pattern or the like and thusin which variances SG1 and SG1′ are large from being incorrectlyidentified as an area in which there is an edge. Note that adetermination as to whether there is an edge extending from upper rightto lower left may be made based on inequality similar to (a) includingonly the mean values without taking into account variances.

As described above with reference to FIGS. 28 to 31, the calculationcircuit 24 determines whether there is an edge in each of fourdirections, the vertical direction, the horizontal direction, thedirection from upper left to lower right, and the direction from upperright to lower left, by determining whether inequalities described aboveare satisfied. Note that the calculation circuit 24 makes thedetermination as to whether there is an edge not only for the greencolor but also red and blue colors.

There is a possibility that inequalities described above are satisfiedin for a plurality of directions of the above four directions. Becausethere can exist only one true edge, and thus only one most dominant edgeis employed and the other edges are discarded.

More specifically, for example, the calculation circuit 24 calculatesthe following value in each of the directions in which edges aredetected|(EGm′−EGm)/(SGm′+SGm)|where m=0 to 3, and the calculation circuit 24 determines mcorresponding to the greatest value. If the value is greatest when m=0,a vertical edge is determined to be dominant. If m=1, a horizontal edgeis determined to be dominant, If m=2, an edge extending in a directionfrom upper left to lower right is determined to be dominant. If m=3, anedge extending in a direction from lower left to lower left isdetermined to be dominant. Note that in the above equation, |x| denotesthe absolute value of x.

Depending on the direction of the edge detected at the position ofinterest, the calculation circuit 24 performs a special processdescribed below on the pixel value Gobs(k, ig, jg) observed at theposition of interest in the edge part.

That is, in the special process, when pixel values Gobs(k, ig, jg) aresubstituted into the spring equation (25) associated with green lightintensities, the pixel value Gobs(k, ig, jg) is changed for the pixelvalue Gobs(k, ig, jg) observed at the position in the edge part.

The changing of the pixel value Gobs(k, ig, jg) to be substituted isperformed, for example, in accordance to a first or second methoddescribed above.

In the first method, when an edge is detected, for example, at aposition of interest denoted by an open circle 83 in FIGS. 28 to 31, thepixel value Gobs(k, ig, jg) at that position is discarded. That is, thepixel value Gobs(k, ig, jg) at the position at which the edge isdetected is not incorporated into the spring equation (25) associatedwith the green light intensity. In other words, a spring that would pullthe light intensity in a wrong direction is removed from the springmodel shown in FIG. 22 (FIG. 23), and thus the value of Lg(I′, J′) canbe determined more accurately (and thus a shaper image can be obtained).

In the second method, the pixel value Gobs(k, ig, jg) at the position ofinterest, for example, denoted by an open circle 83 in FIGS. 28 to 31 isreplaced with a value calculated from pixel values Gobs(k, ig, jg)observed in nine regions A00 to A22 included in the edge judgment regiondefined at the position of interest in the edge detection processdescribed above, and the resultant replaced value is substituted intothe spring equation (25) associated with the green light intensity.

The second method is described in further detail below.

FIG. 32 shows a region A11 including a position of interest denoted byan open circle 83 in a state in which a vertical edge shown in FIG. 28is detected. In FIG. 32, one axis is defined in a X direction in thereference coordinate system, a Y direction in the reference coordinatesystem is defined in a direction perpendicular to the X direction, andan axis perpendicular to both the X direction and the Y directionrepresents pixel values.

In the example shown in FIG. 32, the mean values EG0, EG0′, and EG0″ andthe variances SG0 and SG0′ determined for the edge judgment regionsatisfy the condition EG0′+SG0′<EG0″<EG0−SG0, and thus there is avertical edge.

In this case, the true green light intensity Lg(I−0.5, J−0.5) ata-position (I−0.5, J−0.5) denoted by a solid circle 85 and the truegreen light intensity Lg(I−0.5, J+0.5) at a position (I−0.5, J+0.5)denoted by a solid circle 87 are greater than the pixel value Gobs(k,ig, jg) observed at a position (x, y) of interest denoted by the opencircle 83. The difference between the true green light intensityLg(I−0.5, J−0.5) at the position (I−0.5, J−0.5) denoted by the solidcircle 85 and the pixel value Gobs(k, ig, jg) at the position (x, y)denoted by the open circle 83 and the difference between the true greenlight intensity Lg(I−0.5, J+0.5) at the position (I−0.5, J+0.5) denotedby the solid circle 87 and the pixel value Gobs(k, ig, jg) at theposition (x, y) denoted by the open circle 83 depend on the differencebetween the mean values EG0 and EG0′, that is, EG0−EG0′.

The true green light intensity Lg(I+0.5, J−0.5) at a position (I+0.5,J−0.5) denoted by a solid circle 84 and the true green light intensityLg(I+0.5, J+0.5) at a position (I+0.5, J+0.5) denoted by a solid circle86 are greater than the pixel value Gobs(k, ig, jg) observed at theposition (x, y) of interest denoted by the open circle 83. Thedifference between the true green light intensity Lg(I+0.5, J−0.5) atthe position (I+0.5, J−0.5) denoted by the solid circle 84 and the pixelvalue Gobs(k, ig, jg) at the position (x, y) denoted by the open circle83 and the difference between the true green light intensity Lg(I+0.5,J+0.5) at the position (I+0.5, J+0.5) denoted by the solid circle 86 andthe pixel value Gobs(k, ig, jg) at the position (x, y) denoted by theopen circle 83 depend on the difference between the mean values EG0 andEG0′, that is, EG0−EG0′.

In view of the above, the calculation circuit 24 determines a plane Q1passing through the pixel value Gobs(k, ig, jg) at the position (x, y)denoted by the open circle 83 and having a slope p in the X direction,where p=(EG0−EG0′)/2. The slope p is determined by dividing thedifference between the mean value EG0 of pixel values in the left region91 in FIG. 28 and the mean value EG0′ of pixel values in the rightregion 93 by the center-to-center distance in the X direction betweenthe left region 91 and the right region 92, that is, by 2. Note that theslope p indicates the degree of change in light intensity in a verticaledge part in a direction perpendicular to the edge, that is, the slope pindicates the slope of the edge.

Furthermore, the calculation circuit 24 calculates the value (pixelvalue) at the position (I−0.5, J−0.5) denoted by the solid circle 85 inthe plane Ql. The calculated pixel value is denoted by Gobs0(k, ig, jg).Furthermore, the calculation circuit 24 calculates the value (pixelvalue) at the position (I−0.5, J+0.5) denoted by the solid circle 87 inthe plane Q1 The calculated pixel value is denoted by Gobs1(k, ig, jg).Similarly, the calculation circuit 24 calculates the value (pixel value)at the position (I+0.5, J−0.5) denoted by the solid circle 84 in theplane Q1 and the value (pixel value) at the position (I+0.5, J+0.5)denoted by the solid circle 86 in the plane Q1. The resultant calculatedpixel values are denoted as Gobs2(k, ig, jg) and Gobs3(k, ig, jg),respectively.

Because the plane Q1 has a slope p only in the X direction, the pixelvalue Gobs0(k, ig, jg) and the pixel Gobs1(k, ig, jg) are equal to eachother, and the pixel value Gobs2(k, ig, jg) and the pixel Gobs3(k, ig,jg) are equal to each other.

In the third embodiment described above, as shown in FIG. 22, equation(25) is written based on the spring model representing the equilibriumamong the tree green intensity Lg(I′, J′) at the position (I′, J′) andpixel values Gobs(k, ig, jg) observed in the vicinity of the position(I′, J′). Herein, the pixel values Gobs(k, ig, jg) observed in thevicinity of the position (I′, J′) refer to those pixel values Gobs(k,ig, jg) whose (k, ig, jg) satisfies I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the converted position (x, y) in the reference coordinatesystem obtained by converting, based on conversion parameters (θk, Tkx,Tky, Sk), the position (ig−0.5, jg−0.5) in the coordinate system of thek-th image.

In FIG. 32, for example, when the position (I+0.5, J −0.5) denoted bythe solid circle 84 is selected as the potion (I′, J′) of interest, thepixel value Gobs(k, ig, jg) at the position (x, y) denoted by the opencircle 83 is included in a group of pixel values Gobs(k, ig, jg)observed in the vicinity of the position denoted by the solid circle 84.Therefore, in the third embodiment described above, the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83 issubstituted into equation (25).

In contrast, in the sixth embodiment, instead of the pixel value Gobs(k,ig, jg) at the position (x, y) denoted by the open circle 83, thecalculated value, that is, the pixel value Gobs2(k, ig, jg) at theposition (I+0.5, J−0.5) denoted by the solid circle 84 in the plane Q1is substituted into equation (25).

When the position (I−0.5, J−0.5) denoted by the solid circle 85 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs0(k, ig, jg) at the position (I−0.5,J−0.5) denoted by the solid circle 85 in the plane Q1 is substitutedinto equation (25).

In a case in which the position (I+0.5, J+0.5) denoted by the solidcircle 86 is selected as the potion (I′, J′) of interest, instead of thepixel value Gobs(k, ig, jg) at the position (x, y) denoted by the opencircle 83, the calculated pixel value Gobs3(k, ig, jg) at the position(I+0.5, J+0.5) denoted by the solid circle 86 in the plane Q1 issubstituted into equation (25).

When the position (I−0.5, J+0.5) denoted by the solid circle 87 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs1(k, ig, jg) at the position (I−0.5,J+0.5) denoted by the solid circle 87 in the plane Q1 is substitutedinto equation (25).

As described above, when a pixel value Gobs(k, ig, jg) (such as that atthe position denoted by the open circle 83 in FIG. 32) is observed in avertical edge part (where there is an abrupt change in green lightintensity), the pixel value is changed (corrected) to a pixel value(Gobs0(k, ig, jg), Gobs1(k, ig, jg), Gobs2(k, ig, jg), or Gobs3(k, ig,jg)) depending on the slope p of the edge (the degree of change in pixelvalue), the changed pixel value is substituted into equation (25). As aresult, in the spring model, the object V indicating Lg(I′, J′) shown inFIG. 23 is pulled to a correct position, and thus the obtained valueLg(I′, J′) is more likely to be the true green light intensity (and thusa shaper image can be obtained).

FIG. 33 shows the region A11 including the position of interest denotedby then open circle 83 in a state in which a horizontal edge shown inFIG. 29 is detected. In FIG. 33, one axis is defined in a X direction inthe reference coordinate system, a Y direction in the referencecoordinate system is defined in a direction perpendicular to the Xdirection, and an axis perpendicular to both the X direction and the Ydirection represents pixel values.

In the example shown in FIG. 33, the mean values EG1, EG1′, and EG1″ andthe variances SG1 and SG1′ determined for the edge judgment regionsatisfy the condition EG1′+SG1′<EG1″<EG1−SG1, and thus there is ahorizontal edge.

In this case, the true green light intensity Lg(I+0.5, J−0.5) at aposition (I+0.5, J−0.5) denoted by the solid circle 84 and the truegreen light intensity Lg(I−0.5, J−0.5) at the position (I−0.5, J−0.5)denoted by the solid circle 85 are greater than the pixel value Gobs(k,ig, jg) observed at a position (x, y) of interest denoted by the opencircle 83. The difference between the true green light intensityLg(I+0.5, J−0.5) at the position (I+0.5, J−0.5) denoted by the solidcircle 84 and the pixel value Gobs(k, ig, jg) at the position (x, y)denoted by the open circle 83 and the difference between the true greenlight intensity Lg(I−0.5, J−0.5) at the position (I−0.5, J−0.5) denotedby the solid circle 85 and the pixel value Gobs(k, ig, jg) at theposition (x, y) denoted by the open circle 83 depend on the differencebetween the mean values EG1 and EG1′, that is, EG1−EG1′.

The true green light intensity Lg(I+0.5, J+0.5) at a position (I+0.5,J+0.5) denoted by a solid circle 86 and the true green light intensityLg(I−0.5, J+0.5) at a position (I−0.5, J+0.5) denoted by a solid circle87 are greater than the pixel value Gobs(k, ig, jg) observed at theposition (x, y) of interest denoted by the open circle 83. Thedifference between the true green light intensity Lg(I+0.5, J+0.5) atthe position (I+0.5, J+0.5) denoted by the solid circle 86 and the pixelvalue Gobs(k, ig, jg) at the position (x, y) denoted by the open circle83 and the difference between the true green light intensity Lg(I−0.5,J+0.5) at the position (I−0.5, J+0.5) denoted by the solid circle 87 andthe pixel value Gobs(k, ig, jg) at the position (x, y) denoted by theopen circle 83 depend on the difference between the mean values EG1 andEG1′, that is, EG1−EG1′.

In view of the above, the calculation circuit 24 determines a plane Q2passing through the pixel value Gobs(k, ig, jg) at the position (x, y)denoted by the open circle 83 and having a slope p in the Y direction,where p=(EG1−EG01)/2. The slope p is determined by dividing thedifference between the mean value EG1 of pixel values in the upperregion 101 in FIG. 29 and the mean value EG1′ of pixel values in thelower region 103 by the center-to-center distance in the Y directionbetween the upper region 101 and the lower region 103, that is, by 2.Note that the slope p indicates the degree of change in light intensityin a horizontal edge part in a direction perpendicular to the edge, thatis, the slope p indicates the slope of the edge.

Furthermore, the calculation circuit 24 calculates the value (pixelvalue) at the position (I−0.5, J−0.5) denoted by the solid circle 85 inthe plane Q2. The calculated pixel value is denoted by Gobs0(k, ig, jg).Furthermore, the calculation circuit 24 calculates the value (pixelvalue) at the position (I−0.5, J+0.5) denoted by the solid circle 87 inthe plane Q2. The calculated pixel value is denoted by Gobs1(k, ig, jg).Similarly, the calculation circuit 24 calculates the value (pixel value)at the position (I+0.5, J−0.5) denoted by the solid circle 84 in theplane Q2 and the value (pixel value) at the position (I+0.5, J+0.5)denoted by the solid circle 86 in the plane Q2. The resultant calculatedpixel values are denoted as Gobs2(k, ig, jg) and Gobs3(k, ig, jg),respectively.

Because the plane Q2 has a slope p only in the Y direction, the pixelvalue Gobs0(k, ig, jg) and the pixel Gobs2(k, ig, jg) are equal to eachother, and the pixel value Gobs1(k, ig, jg) and the pixel Gobs3(k, ig,jg) are equal to each other.

For the horizontal edge, substitution is performed in a similar mannerto the vertical edge shown in FIG. 32. That is, when the position(I+0.5, J−0.5) denoted by the solid circle 84 is selected as the potion(I′, J′) of interest, instead of the pixel value Gobs(k, ig, jg) at theposition (x, y) denoted by the open circle 83, the calculated pixelvalue Gobs2(k, ig, jg) at the position (I+0.5, J−0.5) denoted by thesolid circle 84 in the plane Q2 is substituted into equation (25).

When the position (I−0.5, J−0.5) denoted by the solid circle 85 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs0(k, ig, jg) at the position (I−0.5,J−0.5) denoted by the solid circle 85 in the plane Q2 is substitutedinto equation (25).

When the position (I+0.5, J+0.5) denoted by the solid circle 86 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs3(k, ig, jg) at the position (I+0.5,J+0.5) denoted by the solid circle 86 in the plane Q2 is substitutedinto equation (25).

When the position (I−0.5, J+0.5) denoted by the solid circle 87 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs1(k, ig, jg) at the position (x, y)denoted by the solid circle 87 in the plane Q2 is substituted intoequation (25).

As described above, when a pixel value Gobs(k, ig, jg) (such as that atthe position denoted by the open circle 83 in FIG. 33) is observed in ahorizontal edge part (where there is an abrupt change in green lightintensity), the pixel value is changed (corrected) to a pixel value(Gobs0(k, ig, jg), Gobs1(k, ig, jg), Gobs2(k, ig, jg), or Gobs3(k, ig,jg)) depending on the slope p of the edge (the degree of change in pixelvalue), the changed pixel value is substituted into equation (25). As aresult, in the spring model, the object V indicating Lg(I′, J′) shown inFIG. 23 is pulled to a correct position, and thus the obtained valueLg(I′, J′) is more likely to be the true green light intensity (and thusa shaper image can be obtained).

FIG. 34 shows the region A11 including the position of interest denotedby then open circle 83 in a state in which an edge extending in adirection from upper left to lower right shown in FIG. 30 is detected.In FIG. 34, one axis is defined in a X direction in the referencecoordinate system, a Y direction in the reference coordinate system isdefined in a direction perpendicular to the X direction, and an axisperpendicular to both the X direction and the Y direction representspixel values.

In the example shown in FIG. 34, the mean values EG2, EG2′, and EG2″ andthe variances SG2 and SG2′ determined for the edge judgment regionsatisfy the condition EG2′+SG2′<EG2″<EG2′−SG2′, and thus there is anedge extending in the direction from upper left to lower right.

In this case, the true green light intensity Lg(I+0.5, J−0.5) at aposition (I+0.5, J−0.5) denoted by a solid circle 84 is greater than thepixel value Gobs(k, ig, jg) observed at the position (x, y) of interestdenoted by the open circle 83. The difference between the true greenlight intensity Lg(I+0.5, J−0.5) at the position (I+0.5, J−0.5) denotedby the solid circle 84 and the pixel value Gobs(k, ig, jg) at theposition (x, y) denoted by the open circle 83 depend on the differencebetween the mean values EG2 and EG2′, that is, EG2−EG2′.

The true green light intensity Lg(I−0.5, J+0.5) at a position (I−0.5,J+0.5) denoted by a solid circle 87 is greater than the pixel valueGobs(k, ig, jg) observed at the position (x, y) of interest denoted bythe open circle 83. The difference between the true green lightintensity Lg(I−0.5, J+0.5) at the position (I−0.5, J+0.5) denoted by thesolid circle 87 and the pixel value Gobs(k, ig, jg) at the position (x,y) denoted by the open circle 83 depend on the difference between themean values EG2′ and EG2, that is, EG2′−EG2.

In view of the above, the calculation circuit 24 determines a plane Q3passing through the pixel value Gobs(k, ig, jg) at the position (x, y)of interest denoted by the open circle 83 and having a slope p in adiagonal direction of the region A11, that is, in a direction from theposition (I+0.5, J−0.5) denoted by the solid circle 84 to the position(I−0.5, J+0.5) denoted by the solid circle 87, where p=(EG2′−EG2)/2√2.The slope p is determined by dividing the difference between the meanvalue EG2′ of pixel values in the upper right region 113 in FIG. 30 andthe mean value EG2 of pixel values in the lower left region 111 by 242,that is, the center-to-center distance in a diagonal direction from theposition (I+0.5, J−0.5) denoted by the solid circle 84 to the position(I−0.5, J+0.5) denoted by the solid circle 87 between the region A20 inthe upper right region 113 and the region A02 in the lower left region111. Note that the slope p of an edge part extending in the directionfrom upper left to lower right indicates the degree of change in lightintensity in a direction perpendicular to the edge, that is, the slope pindicates the slope of the edge.

Furthermore, the calculation circuit 24 calculates the value (pixelvalue) at the position (I−0.5, J−0.5) denoted by the solid circle 85 inthe plane Q3. The calculated pixel value is denoted by Gobs0(k, ig, jg).The calculation circuit 24 also calculates the value (pixel value) atthe position (I−0.5, J+0.5) denoted by the solid circle 87 in the planeQ3. The calculated pixel value is denoted by Gobs1(k, ig, jg).Similarly, the calculation circuit 24 calculates the value (pixel value)at the position (I+0.5, J−0.5) denoted by the solid circle 84 in theplane Q3 and the value (pixel value) at the position (I+0.5, J+0.5)denoted by the solid circle 86 in the plane Q3. The resultant calculatedpixel values are denoted as Gobs2(k, ig, jg) and Gobs3(k, ig, jg),respectively.

Because the plane Q3 has a slope p only in the diagonal direction of theregion A11, that is, in the direction from the position (I+0.5, J−0.5)denoted by the solid circle 84 to the position (I−0.5, J+0.5) denoted bythe solid circle 87, the pixel value Gobs0(k, ig, jg) and the pixelGobs3(k, ig, jg) are equal to each other.

For the edge extending in the direction from upper left to lower right,substitution is performed in a similar manner to the vertical edge shownin FIG. 32. That is, when the position (I+0.5, J−0.5) denoted by thesolid circle 84 is selected as the potion (I′, J′) of interest, insteadof the pixel value Gobs(k, ig, jg) at the position (x, y) denoted by theopen circle 83, the calculated pixel value Gobs2(k, ig, jg) at theposition (I+0.5, J−0.5) denoted by the solid circle 84 in the plane Q3is substituted into equation (25).

When the position (I−0.5, J−0.5) denoted by the solid circle 85 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs0(k, ig, jg) at the position (I−0.5,J−0.5) denoted by the solid circle 85 in the plane Q3 is substitutedinto equation (25).

When the position (I+0.5, J+0.5) denoted by the solid circle 86 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs3(k, ig, jg) at the position (I+0.5,J+0.5) denoted by the solid circle 86 in the plane Q3 is substitutedinto equation (25).

When the position (I−0.5, J+0.5) denoted by the solid circle 87 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs1(k, ig, jg) at the position (I−0.5,J+0.5) denoted by the solid circle 87 in the plane Q3 is substitutedinto equation (25).

As described above, when a pixel value Gobs(k, ig, jg) (such as that atthe position denoted by the open circle 83 in FIG. 34) is observed in anedge part extending in the direction from upper left to lower right(where there is an abrupt change in green light intensity), the pixelvalue is changed (corrected) to a pixel value (Gobs0(k, ig, jg),Gobs1(k, ig, jg), Gobs2(k, ig, jg), or Gobs3(k, ig, jg)) depending onthe slope p of the edge (the degree of change in pixel value), thechanged pixel value is substituted into equation (25). As a result, inthe spring model, the object V indicating Lg(I′, J′) shown in FIG. 23 ispulled to a correct position, and thus the obtained value Lg(I′, J′) ismore likely to be the true green light intensity (and thus a shaperimage can be obtained).

FIG. 35 shows the region A11 including the position of interest denotedby then open circle 83 in a state in which an edge extending in adirection from upper right to lower left shown in FIG. 31 is detected.In FIG. 34, one axis is defined in a X direction in the referencecoordinate system, a Y direction in the reference coordinate system isdefined in a direction perpendicular to the X direction, and an axisperpendicular to both the X direction and the Y direction representspixel values.

In the example shown in FIG. 35, the mean values EG3, EG3′, and EG3″ andthe variances SG3 and SG3′ determined for the edge judgment regionsatisfy the condition EG3′+SG3′<EG3″<EG3- SG3, and thus there is an edgeextending in the direction from upper right to lower left.

In this case, the true green light intensity Lg(I−0.5, J−0.5) at theposition (I−0.5, J−0.5) denoted by the solid circle 85 is greater thanthe pixel value Gobs(k, ig, jg) observed at the position (x, y) ofinterest denoted by the open circle 83. The difference between the truegreen light intensity Lg(I−0.5, J−0.5) at the position (I−0.5, J−0.5)denoted by the solid circle 85 and the pixel value Gobs(k, ig, jg) atthe position (x, y) denoted by the open circle 83 depend on thedifference between the mean values EG3 and EG3′, that is, EG3−EG3′.

The true green light intensity Lg(I+0.5, J+0.5) at the position (I+0.5,J+0.5) denoted by the solid circle 86 is greater than the pixel valueGobs(k, ig, jg) observed at the position (x, y) of interest denoted bythe open circle 83. The difference between the true green lightintensity Lg(I+0.5, J+0.5) at the position (I+0.5, J+0.5) denoted by thesolid circle 86 and the pixel value Gobs(k, ig, jg) at the position (x,y) denoted by the open circle 83 depend on the difference between themean values EG3 and EG3′, that is, EG3−EG3′.

In view of the above, the calculation circuit 24 determines a plane Q4passing through the pixel value Gobs(k, ig, jg) at the position (x, y)of interest denoted by the open circle 83 and having a slope p in adiagonal direction of the region A11, that is, in a direction from theposition (I−0.5, J−0.5) denoted by the solid circle 85 to the position(I+0.5, J+0.5) denoted by the solid circle 86, where p=(EG3−EG3′)/2√2.The slope p is determined by dividing the difference between the meanvalue EG3 of pixel values in the upper left region 121 in FIG. 31 andthe mean value EG3′ of pixel values in the lower right region 123 by2√2, that is, the center-to-center distance in a diagonal direction fromthe position (I−0.5, J−0.5) denoted by the solid circle 85 to theposition (I+0.5, J+0.5) denoted by the solid circle 86 between theregion A00 in the upper left region 121 and the region A22 in the lowerright region 123. Note that the slope p of an edge part extending in thedirection from upper right to lower left indicates the degree of changein light intensity in a direction perpendicular to the edge, that is,the slope p indicates the slope of the edge.

Furthermore, the calculation circuit 24 calculates the value (pixelvalue) at the position (I−0.5, J−0.5) denoted by the solid circle 85 inthe plane Q4. The calculated pixel value is denoted by Gobs0(k, ig, jg).The calculation circuit 24 also calculates the value (pixel value) atthe position (I−0.5, J+0.5) denoted by the solid circle 87 in the planeQ4. The calculated pixel value is denoted by Gobs1(k, ig, jg).Similarly, the calculation circuit 24 calculates the value (pixel value)at the position (I+0.5, J−0.5) denoted by the solid circle 84 in theplane Q4 and the value (pixel value) at the position (I+0.5, J+0.5)denoted by the solid circle 86 in the plane Q4. The resultant calculatedpixel values are denoted as Gobs2(k, ig, jg) and Gobs3(k, ig, jg),respectively.

Because the plane Q4 has a slope p only in the diagonal direction of theregion A11, that is, in the direction from the position (I−0.5, J−0.5)denoted by the solid circle 85 to the position (I+0.5, J+0.5) denoted bythe solid circle 86, the pixel value Gobs1(k, ig, jg) and the pixelGobs2(k, ig, jg) are equal to each other.

For the edge extending in the direction from upper right to lower left,substitution is performed in a similar manner to the vertical edge shownin FIG. 32. That is, when the position (I+0.5, J−0.5) denoted by thesolid circle 84 is selected as the potion (I′, J′) of interest, insteadof the pixel value Gobs(k, ig, jg) at the position (x, y) denoted by theopen circle 83, the calculated pixel value Gobs2(k, ig, jg) at theposition (I+0.5, J−0.5) denoted by the solid circle 84 in the plane Q4is substituted into equation (25).

When the position (I−0.5, J−0.5) denoted by the solid circle 85 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs0(k, ig, jg) at the position (I−0.5,J−0.5) denoted by the solid circle 85 in the plane Q4 is substitutedinto equation (25).

When the position (I+0.5, J+0.5) denoted by the solid circle 86 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs3(k, ig, jg) at the position (I+0.5,J+0.5) denoted by the solid circle 86 in the plane Q4 is substitutedinto equation (25).

When the position (I−0.5, J+0.5) denoted by the solid circle 87 isselected as the potion (I′, J′) of interest, instead of the pixel valueGobs(k, ig, jg) at the position (x, y) denoted by the open circle 83,the calculated pixel value Gobs1(k, ig, jg) at the position (I−0.5,J+0.5) denoted by the solid circle 87 in the plane Q4 is substitutedinto equation (25).

As described above, when a pixel value Gobs(k, ig, jg) (such as that atthe position denoted by the open circle 83 in FIG. 35) is observed in anedge part extending in the direction from upper right to lower left(where there is an abrupt change in green light intensity), the pixelvalue is changed (corrected) to a pixel value (Gobs0(k, ig, jg),Gobs1(k, ig, jg), Gobs2(k, ig, jg), or Gobs3(k, ig, jg)) depending onthe slope p of the edge (the degree of change in pixel value), thechanged pixel value is substituted into equation (25). As a result, inthe spring model, the object V indicating Lg(I′, J′) shown in FIG. 23 ispulled to a correct position, and thus the obtained value Lg(I′, J′) ismore likely to be the true green light intensity (and thus a shaperimage can be obtained).

From the above discussion, in this sixth embodiment, a spring equationassociated with the green light intensity corresponding to equation (25)according to the third embodiment is given by equation (34) shown below.Note that the true red light intensity Lr(I′, J′) and the true bluelight intensity Lb(I′, J′) can be determined in a similar manner to thetrue green light intensity Lg(I′, J′). $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Gobs}^{\prime}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (34)\end{matrix}$

For the given position (I′, J′), the sum E in equation (34) is taken forall combinations (k, ig, jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1with respect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the position (ig−0.5, jg−0.5) on the k-th image.

Note that when one of a vertical edge, a horizontal edge, an edgeextending from upper left to lower right, and an edge extending fromupper right to lower left is detected at a position of a pixel valueGobs(k, ig, jg) of one of combinations of integers (k, ig, jg) includedin equation (34), a calculated pixel value at a position (I′, J′) in aplane passing through that pixel value Gobs(k, ig, jg) of (k, ig, jg)and having the slope p equal to the slope of the edge is substitutedinto Gobs′(k, ig, jg) of equation (34) instead of the pixel valueGobs(k, ig, jg) of (k, ig, jg). When none of a vertical edge, ahorizontal edge, an edge extending from upper left to lower right, andan edge extending from upper right to lower left is detected at aposition of a pixel value Gobs(k, ig, jg) of any of combinations ofintegers (k, ig, jg) included in equation (34), the pixel value Gobs(k,ig, jg) of (k, ig, jg) is directly substituted into Gobs′(k, ig, jg) inequation (34).

Furthermore, in this sixth embodiment, a spring equation associated withthe green light intensity and a spring equation associated with the bluelight intensity corresponding to equations (26) and (27)according to thethird embodiment are is given by equations (35) and (36) shown below.$\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Robs}^{\prime}\left( {k,{ir},{jr}} \right)} - {{Lr}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (35)\end{matrix}$

For the given position (I′, J′), the sum E in equation (35) is taken forall combinations (k, ir, jr) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1with respect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the position (ir−0.5, jr−0.5) on the k-th image.

Note that when one of a vertical edge, a horizontal edge, an edgeextending from upper left to lower right, and an edge extending fromupper right to lower left is detected at a position of a pixel valueRobs(k, ir, jr) of one of combinations of integers (k, ir, jr) includedin equation (35), a calculated pixel value at a position (I′, J′) in aplane passing through that pixel value Robs(k, ir, jr) of (k, ir, jr)and having the slope p equal to the slope of the edge is substitutedinto Robs′(k, ir, jr) in equation (35) instead of the pixel valueRobs(k, ir, jr) of (k, ir, jr). When none of a vertical edge, ahorizontal edge, an edge extending from upper left to lower right, andan edge extending from upper right to lower left is detected at aposition of a pixel value Gobs(k, ig, jg) of any of combinations ofintegers (k, ig, jg) included in equation (35), the pixel value Robs(k,ir, jr) of (k, ir, jr) is directly substituted into Robs′(k, ir, jr) inequation (35). $\begin{matrix}{{\sum\left( {\frac{\sqrt{2} - {F\left( {\left( {x,y} \right),\left( {I^{\prime},J^{\prime}} \right)} \right)}}{\left( {{Mk} \times E} \right)} \times \left\{ {{{Bobs}^{\prime}\left( {k,{ib},{jb}} \right)} - {{Lb}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (36)\end{matrix}$

For the given position (I′, J′), the sum I in equation (36) is taken forall combinations (k, ib, jb) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1with respect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the position (ib−0.5, jb−0.5) on the k-th image.

Note that when one of a vertical edge, a horizontal edge, an edgeextending from upper left to lower right, and an edge extending fromupper right to lower left is detected at a position of a pixel valueBobs(k, ib, jb) of one of combinations of integers (k, ib, jb) includedin equation (36), a calculated pixel value at a position (I′, J′) in aplane passing through that pixel value Bobs(k, ib, jb) of (k, ib, jb)and having the slope equal to the slope of the edge is substituted intoBobs′(k, ib, jb) of equation (36) instead of the pixel value Bobs(k, ib,jb) of (k, ib, jb). When none of a vertical edge, a horizontal edge, anedge extending from upper left to lower right, and an edge extendingfrom upper right to lower left is detected at a position of a pixelvalue Gobs(k, ig, jg) of any of combinations of integers (k, ig, jg)included in equation (36), the pixel value Bobs(k, ib, jb) of (k, ib,jb) is directly substituted into Bobs′(k, ib, jb) in equation (36).

Now, referring to a flow chart shown in FIG. 36, a correct imageestimation process in step S4 shown in FIG. 2 according to a sixthembodiment of the present invention is described.

First, in step S141, the calculation circuit 24 selects a position (I′,J′) in the reference coordinate system (hereinafter, such a selectedposition will be referred to a position (I′, J′) of interest). Note thatthe position (I′, J′) of interest is at the center (i−0.5, j−0.5) of a(i-th, j-th) pixel of the first image taken as the reference image.

Thereafter, the process proceeds from step S141 to step S142. In stepS142, the calculation circuit 24 determines all combinations of integers(k, ig, jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel that senses a greencomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS143.

In step S143, the calculation circuit 24 determines whether there is avertical edge, a horizontal edge, an edge extending from upper left tolower right, or an edge extending from upper right to lower left, foreach of combinations of integers (k, ig, jg) determined in step S142. Ifit is determined in step S143 that there is an edge extending in one ofthe four directions, the process proceeds to step S144. In step S144,for each combination (k, ig, jg) corresponding to the position at whichthe edge was detected, the calculation circuit 24 produces (determines)a plane passing through the pixel Gobs(k, ig, jg) and having a slopeequal to the slope p of the edge. The calculation circuit 24 thencalculates the value (pixel value) at the position (I′, J′) of interestin the plane. Thereafter, the process proceeds from step S144 to stepS145.

In the case in which it is determined in step S143 that there is no edgein any of the four directions, the process proceeds to step Sl45. Notethat the process also proceeds to step S145 after completion of stepS144. In step S145, the calculation circuit 24 produces a springequation in the form (34) for green light intensity, using allcombinations of (k, ig, ig) determined in step S142. Thereafter, theprocess proceeds to step S146. For a pixel value Gobs(k, ig, jg) at aposition corresponding to (k, ig, jg) in the reference coordinate systemat which an edge was detected in step S143, the calculation circuit 24employs the value (pixel value) at the position (I′, J′) in the planedetermined in step S144 as the pixel value Gobs(k, ig, ig) of (k, ig,jg) at which the edge was detected, and the calculation circuit 24substitutes it into Gobs′(k, ig, jg) in equation (34). However, for thepixel value Gobs(k, ig, ig), if it is determined in step S143 that thereis no edge in any of the four directions at the position in thereference coordinate system corresponding to (k, ig, jg), thecalculation circuit 24 directly substitutes the pixel value Gobs (k, ig,jg) of (k, ig, ig) into Gobs′(k, ig, jg) in equation (34).

In step S146, the calculation circuit 24 determines all combinations ofintegers (k, ir, jr) that satisfy I′−1≦x <I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ir−0.5, jr−0.5) of the pixel that. senses ared component of the k-th image, for the position (I′, J′) of interestof each of first to N-th images. Thereafter, the process proceeds tostep S147.

In step S147, the calculation circuit 24 determines whether there is avertical edge, a horizontal edge, an edge extending from upper left tolower right, or an edge extending from upper right to lower left, foreach of combinations of integers (k, ir, jr) determined in step S146. Ifit is determined in step S147 that there is an edge extending in one ofthe four directions, the process proceeds to step S148. In step S148,for each combination (k, ir, jr) corresponding to the position at whichthe edge was detected, the calculation circuit 24 produces (determines)a plane passing through the pixel Robs(k, ir, jr) and having a slopeequal to the slope p of the edge. The calculation circuit 24 thencalculates the value (pixel value) at the position (I′, J′) of interestin the plane. Thereafter, the process proceeds from step S148 to stepS149.

In the case in which it is determined in step S147 that there is no edgein any of the four directions, the process proceeds to step S149. Notethat the process also proceeds to step S149 after completion of stepS148. In step S149, the calculation circuit 24 produces a springequation in the form (35) for red light intensity, using allcombinations of (k, ir, jr) determined in step S146. Thereafter, theprocess proceeds to step S150. For a pixel value Robs(k, ir, jr) at aposition corresponding to (k, ir, jr) in the reference coordinate systemat which an edge was detected in step S147, the calculation circuit 24employs the value (pixel value) at the position (I′, J′) in the planedetermined in step S148 as the pixel value Robs(k, ir, jr) of (k, ir,jr) at which the edge was detected, and the calculation circuit 24substitutes it into Robs′(k, ir, jr) in equation (35). However, for thepixel value Robs(k, ir, jr), if it is determined in step S147 that thereis no edge in any of the four directions at the position in thereference coordinate system corresponding to (k, ir, jr), thecalculation circuit 24 directly substitutes the pixel value Robs(k, ir,jr) of (k, ir, jr) into Robs′(k, ir, jr) in equation (35).

In step S150, the calculation circuit 24 determines all combinations ofintegers (k, ib, jb) that satisfy I′−1≦x <I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ib−0.5, jb−0.5) of the pixel that senses ablue component of the k-th image, for the position (I′, J′) of interestof each of first to N-th images. Thereafter, the process proceeds tostep S151.

In step S151, the calculation circuit 24 determines whether there is avertical edge, a horizontal edge, an edge extending from upper left tolower right, or an edge extending from upper right to lower left, foreach of combinations of integers (k, ib, jb) determined in step S152. Ifit is determined in step S151 that there is an edge extending in one ofthe four directions, the process proceeds to step S152. In step S152,for each combination (k, ib, jb) corresponding to the position at whichthe edge was detected, the calculation circuit 24 produces (determines)a plane passing through the pixel Bobs(k, ib, jb) and having a slopeequal to the slope p of the edge. The calculation circuit 24 thencalculates the value (pixel value) at the position (I′, J′) of interestin the plane. Thereafter, the process proceeds from step S152 to stepS153.

In the case in which it is determined in step S151 that there is no edgein any of the four directions, the process proceeds to step S153. Notethat the process also proceeds to step S153 after completion of stepS152. In step S153, the calculation circuit 24 produces a springequation in the form (36) for blue light intensity, using allcombinations of (k, ib, jb) determined in step S150. Thereafter, theprocess proceeds to step S154. For a pixel value Bobs(k, ib, jb) at aposition corresponding to (k, ib, jb) in the reference coordinate systemat which an edge was detected in step S151, employs the value (pixelvalue) at the position (I′, J′) in the plane determined in step S152 asthe pixel value Bobs(k, ib, jb) of (k, ib, jb) at which the edge wasdetected, and the calculation circuit 24 substitutes it into Gobs′(k,ib, jb) in equation (36). However, for the pixel value Bobs(k, ib, jb),if it is determined in step S151 that there is no edge in any of thefour directions at the position in the reference coordinate systemcorresponding to (k, ib, jb), the calculation circuit 24 directlysubstitutes the pixel value Bobs(k, ib, jb) of (k, ib, jb) into Bobs′(k,ib, jb) in equation (36).

In step S154, the calculation circuit 24 solves the spring equation (34)associated with the red light intensity determined in step S145, thespring equation (35) associated with the red light intensity determinedin step S149, and the spring equation (36) associated with the bluelight intensity determined in step S153, which are all in the form of alinear equation, thereby determining the true green light intensityLg(I′, J′), the true red light intensity Lr(I′, J′), and the true bluelight intensity Lb(I′, J′) at the position (I′, J′) of interest.Thereafter, the process proceeds to step S155.

In step S155, the calculation circuit 24 determines whether theabove-described process is completed for all possible positions (I′,J′), that is, whether the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, J′), and the true blue light intensityLb(I′, J′) have been determined by taking the center position of each ofall pixels of the first image as the position (I′, J′) of interest.

If it is determined in step S155 that the process is not completed forall possible positions (I′, J′) of interest, the process returns to stepS141 to repeat steps S141 to S155. That is, the calculation circuit 24selects, as a new position of interest, a position (I′, J′) that has notyet been selected as the position of interest, and determines the truegreen light intensity Lg(I′, J′), the true red light intensity Lr(I′,J′), and the true blue light intensity Lb(I′, J′) at the position (I′,J′) of interest.

On the other hand in the case in which it is determined in step S155that the process is completed by taking all possible position (I′, J′)as the position of interest, the process proceeds to step S156. In stepS156, the calculation circuit 24 estimates a correct image (signal) fromthe true green light intensity Lg(I′, J′), the true red light intensityLr(I′, J′), and the true blue light intensity Lb(I′, Ji) determined instep S154. The resultant image is supplied as an output image to the D/Aconverter 9 or the codec 12, and the current process is exited. Forexample, for the (i-th, j-th) pixel, the calculation circuit 24estimates a correct image signal thereof from the true green lightintensity Lg(i−0.5, j−0.5) determined as the green value (G signal) instep S154, the true red light intensity Lr(i−0.5, j−0.5) determined asthe red value (R signal) in step S154, and the true blue light intensityLb(i−0.5, j−0.5) determined as the blue value (B signal) in step S154.The calculation circuit 24 estimates the output image by estimating theimage signal for all pixels whose center is located at the selectedposition (I′, J′).

In the sixth embodiment, as described above, for a pixel value observedin an edge part, the pixel value is changed depending on the slope ofthe edge (the degree of change in pixel value), and the resultantchanged pixel value is substituted into a corresponding spring equation.This makes it possible to obtain an image more likely to be a correctimage. That is, a sharper image can be obtained.

In the example described above, a pixel value located in an edge part isreplaced with a pixel value at the position (I′, J′) of interestdepending on the slope (steepness) of the edge. Alternatively, forexample, the pixel value in the edge part may be directly used, and theweight is modified depending on the distance to the position (I′, J′) ofinterest so that the pixel value located in the edge part does not havea significant effect on the spring equation.

In the example described above, the slopes of planes Q1 to Q4 aredetermined using pixel values observed within nine regions (that is, aregion with a size of 3×3 pixels) centered at the position (x, y) in thereference coordinate system. Alternatively, the slopes of planes Q1 toQ4 may be determined using pixel values observed in regions selecteddifferently. For example, sixteen regions (tat is, a region with a sizeof 4×4 pixels) may be employed.

Now, a correct image estimation process performed by the signalprocessor 7 according to a seventh embodiment of the present inventionis described. The seventh embodiment is a modification to the thirdembodiment described earlier with reference to FIG. 24.

That is, in the seventh embodiment, in addition to the spring equations(25), (26), and (27), the constraint associated with the colorcorrelation among R, G, and B signals is imposed on the process ofdetermining the green light intensity Lg(x, y), the true red lightintensity Lr(x, y), and the true blue light intensity Lb(x, y).

In a local area of an image, there is a color correlation among colorsof the true green light intensity Lg(x, y), the true red light intensityLr(x, y), and the true blue light intensity Lb(x, y) corresponding tolight originating from a subject and being incident on the imagingdevice 4. Therefore, by imposing the color correlation constraint on theestimation of a true image, it becomes possible to produce a sharperlow-noise image more likely to be the correct image.

Referring to FIGS. 37 and 38, a specific method of determining a colorcorrelation constraint is described below. In the following descriptionreferring to FIGS. 37 and 38, by way of example, the correlation betweengreen and red colors will be discussed.

By way of example, the discussion will be made for a green pixelG(jg−1)(ig−1) at a (ig-th, jg-th) position of a k′-th image shown in alower left part of FIG. 37 a red pixel G(jg−1)(ig−1) at a (ir-th, jr-th)position of a k″-th image shown in a lower right part of FIG. 37.

The calculation circuit 24 converts positions of the green pixelG(jg−1)(ig−1) of the k′-th image and the red pixel R(jr−1)(ir−1) of thek″-th image into positions in the coordinate system of the first imageon the upper side of FIG. 37 by using conversion parameters (θk′, Tk′x,Tk′y, Sk′) and (θk″, Tk″x, Tkily, Sk″) in a similar manner to thataccording to the third embodiment.

The calculation circuit 24 then calculates the distance between thegreen pixel G(jg−1)(ig−1) of the k′-th image at the position convertedinto the coordinate system of the first image and the red pixelR(jr−1)(ir−1) of the k′-th image at the position converted into thecoordinate system of the first image. Furthermore, the calculationcircuit 24 determines whether these two pixels can be regarded aslocating at the same position by determining whether the calculateddistance is less than a maximum allowable value (threshold value) delta(set to, for example, 0.25 times the pixel size).

When positions of pixels G(jg−1)(ig−1) and R(jr−1)(ir−1) arerespectively given as positions (ig, jg) and (ir, jr), if the position(ig, jg) of the pixel G(jg−1)(ig−1) in the coordinate system of thek′-th image is denoted as G(c), the position (ir, jr) of the pixelR(jg−1)(ig−1) in the coordinate system of the k″-th image is denoted asR(c), the position (ig, jg) of the pixel G(jg−1)(ig−1) in the coordinatesystem of the first image is denoted as Gc(k′), and the position (ir,jr) of the pixel R(jg−1)(ig−1) in the coordinate system of the firstimage is denoted as Rc(k′), then a constraint requiring that thedistance between point G_(c(k′)) and point R_(c(k″)) should be withinthe allowable value delta is given by formula (37) shown below.$\begin{matrix}{{{Dis}\left\lbrack {G_{c{(k^{\prime})}},R_{c{(k^{\prime\prime})}}} \right\rbrack} = {{{Dis}\left\lbrack {{{{{Sk}^{\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ig} \\{jg}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime}x} \\{{Tk}^{\prime}y}\end{pmatrix}},{{{{Sk}^{\prime\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ir} \\{jr}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime\prime}x} \\{{Tk}^{\prime\prime}y}\end{pmatrix}}} \right\rbrack} \leq {delta}}} & (37)\end{matrix}$

Formula (37) is referred to as a distance constraint formula. In formula(23), Dis[Gc(k′), RC(k′,)] denotes the distance between point GC(k.) andpoint RC(k.′). The positions of point G_(c(k′))) and point R_(c(k′)) areobtained by performing the affine transformation on positions (ig, jg)and (ir, jr) using conversion parameters (θk′, Tk′x, Tk′y, Sk′) and(θk″, Tk″x, Tk″y, Sk″).

The calculation circuit 24 determines whether there are. a green pixelG(jg−1)(ig−1) of the k′-th image and a red pixel; R(jr−1)(ir−1) of thek″-th image at positions that can be regarded as being the same withinan allowable margin delta in a small region (I′+dX, J′+dY) centered at aposition (I′, J′) in the coordinate system of the first image, that is,in a rectangular region whose corners are at (I′−dX, J′−dY), (I′−dX,J′+dY), (I′+dX, J′−dY), and (I′+dX, J′+dY), wherein dX and dY are valuesthat define the region in the vicinity of the position (x, y) and theyare set to be equal to, for example, 2 times the size of a pixel in theX direction and Y direction, respectively.

In other words, the calculation circuit 24 determines (kl, ig, jg) and(kit, ir, jr) that satisfy formula (37) within the small region (I′+dX,J′+dY) centered at the position (I′, J′) of interest in the coordinatesystem of the first image, that is, in the rectangular region whosecorners are at (I′−dX, J′−dY), (I′−dX, J′+dY), (I′+dX, J′−dY), and(I′+dX, J′+dY).

The calculation circuit 24 then acquires pixel values Gobs(kl, ig, jg)and Robs(k″, ir, jr) respectively corresponding to (k′, ig, jg) and (k′,ir, jr) determined.

The calculation circuit 24 determines (k′, ig, jg) and (k″, ir, jr) thatsatisfy formula (37) for all possible combinations of k′ and k″ withinthe range form 1 to N.

In general, a plurality of combinations of (k′, ig, jg) and (k″, ir, jr)is detected, and the calculation circuit 24 plots pixel values Gobs(k′,ig, jg) and Robs(k″, ir, jr) corresponding to the detected (k′, ig, jg)and (k″, ir, jr) in the GR space in which the horizontal axis representsthe G signal (Gobs(k′, ig, jg)) and the vertical axis represents the Rsignal (Robs(k″, ir, jr)), as shown in FIG. 38.

FIG. 38 shows the GR space in which sets of pixel values Gobs(k′, ig,jg) and Robs(k″, ir, jr) satisfying formula (37) are plotted.

In FIG. 38, each x-like mark indicates a pair of pixel values Gobs(k′,ig, jg) and Robs(k′ , ir, jr) corresponding to (k′, ig, jg) and (k″, ir,jr) detected by the calculation circuit 24, that is, a pair of pixelvalues Gobs(k′, ig, jg) and Robs(k″, ir, jr) satisfying formula (37).

The plots shown in FIG. 38 indicate that there is a correlation betweenthe true green light intensity Lg(I′, J′) and the true red lightintensity Lr(I′, J′) to be determined in the vicinity of the position(I′, J′) of interest.

Thus, in the seventh embodiment, in addition to spring constraints givenby equations (25) to (27) in the third embodiment, the correlationbetween green and red colors shown in FIG. 38 is added as a constrain.

The calculation circuit 24 performs principal component analysis for theplurality of points plotted in the GR space shown in FIG. 38, whereineach plotted point corresponds to a pair of pixel values Gobs(k′, ig,jg) and Robs(k″, ir, jr) that the calculation circuit 24 has detected assatisfying the distance constraint (37).

The calculation circuit 24 then determines a principal direction as aresult of the principal component analysis and determines a lineQg×G+Qr×R=Qc representing the determined direction (axis) where Qg, Qr,and Qc are constants associated with the line in the GR space, and G andR are variables associated with the G and R signals in the GR space.Furthermore, the calculation circuit 24 determines the variance in adirection perpendicular to the principal direction.

Herein, a spring with a natural length of 0 and with a spring constant(strength) H is introduced. The spring constant may be an arbitrarymonotonically decreasing function of the variance of the componentperpendicular to the direction of the principal component determined inthe GR space. For example, H=(1/variance) may be used. In any case, thestrength of the spring (spring constant) increases with decreasingvariance.

One end of the spring with the spring constant H is connected to point(Lg(I′, J′), Lr(I′, J′)) in the GR space, and the other end of thespring is connected such that it can movable to an arbitrary point onthe line Qg×G+Qr x R=Qc. As a result, the point (Lg(I′, J′), Lr(I′, J′))is pulled in a direction of the perpendicular line from the point(Lg(I′, J′), Lr(I′, J′)) to the line Qg×G+Qr×R =Qc. That is, the point(Lg(I′, J′), Lr(I′, J′)) is pulled such that the color correlationconstraint (between the G and R signals) is satisfied.

Herein, if the distance between the point (Lg(I′, J′), Lr(I′, J′)) andthe line Qg×G+Qr×R=Qc extending in the direction of the main componentis represented by u, then the pulling force of the spring with thespring constant H exerted on the point (Lg(I′, J′), Lr(I′, J′)) is givenby H×u (hereinafter, this force will be referred to as GR spring force).The GR spring force H×u in the GR space is broken down into a G signalcomponent and a R signal component, and these components arerespectively added to the left-hand sides of equations (25) and (26)such that the color correlation is taken into account in equations (25)and (26).

The color correlation between the green (G) signal and the B (Blue)signal is dealt with as follows.

As in the case of the color correlation between the green and redsignals described above with reference to FIG. 37, the calculationcircuit 24 converts positions of the green pixel G(jg−1)(ig−1) of thek′-th image and the blue pixel B(jb−1)(ib−1) of the k′″-th image intopositions in the coordinate system of the first image by usingconversion parameters (θk′, Tk′x, Tk′y, Sk′) and (θk′″, Tk′″x, Tk′″y,Sk′″).

The calculation circuit 24 then calculates the distance between thegreen pixel G(jg−1)(ig−1) of the k′-th image at the position convertedinto the coordinate system of the first image and the blue pixelB(jb−1)(ib−1) of the k′″-th image at the position converted into thecoordinate system of the first image. Furthermore, the calculationcircuit 24 determines whether these two pixels can be regarded aslocating at the same position by determining whether the calculateddistance is less than a maximum allowable value (threshold value) delta.

When positions of pixels G(jg−1)(ig−1) and B(jb−1)(ib−1) arerespectively given as positions (ig, jg) and (ib, jb), if the position(ig, jg) of the pixel G(jg−1)(ig−1) in the coordinate system of thekl-th image is denoted as point G(c), the position (ib, jb) of the pixelB(jb−1)(ib−1) in the coordinate system of the k′″-th image is denoted aspoint B(c), the position (ig, jg) of the pixel G(jg−1)(ig−1) in thecoordinate system of the first image is denoted as G_(c(k′)), and theposition (ib, jb) of the pixel B(jb−1)(ib−1) in the coordinate system ofthe first image is denoted as BC(k′″), then a constraint requiring thatthe distance between point Gc(k′) and point B_(c(k′″)) should be withinthe allowable value delta is given by formula (38) shown below.$\begin{matrix}{{{Dis}\left\lbrack {G_{c{(k^{\prime})}},B_{c{(k^{\prime\prime\prime})}}} \right\rbrack} = {{{Dis}\left\lbrack {{{{{Sk}^{\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ig} \\{jg}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime}x} \\{{Tk}^{\prime}y}\end{pmatrix}},{{{{Sk}^{\prime\prime\prime}\begin{pmatrix}{\cos\left( {\theta\quad k^{\prime\prime\prime}} \right)} & {- {\sin\left( {\theta\quad k^{\prime\prime\prime}} \right)}} \\{\sin\left( {\theta\quad k^{\prime\prime\prime}} \right)} & {\cos\left( {\theta\quad k^{\prime\prime\prime}} \right)}\end{pmatrix}}\begin{pmatrix}{ib} \\{jb}\end{pmatrix}} + \begin{pmatrix}{{Tk}^{\prime\prime\prime}x} \\{{Tk}^{\prime\prime\prime}y}\end{pmatrix}}} \right\rbrack} \leq {delta}}} & (38)\end{matrix}$

Formula (38) is referred to as a distance constraint formula. In formula(23), Dis[G_(c(k′)), B_(c(k′″))] denotes the distance between pointG_(c(k′)) and point BC(k .). The positions of point G_(c(k′))) and pointBC(k′″.) are obtained by performing the affine transformation onpositions (ig, jg) and (ib, jb) using conversion parameters (θk′, Tk′x,Tk′y, Sk′) and (θk′″, Tk′″x, Tk″y, Sk′″).

The calculation circuit 24 determines whether there are a green pixelG(jg−1)(ig−1) of the k′-th image and a blue pixel B(jb−1)(ib−1) of thek′″-th image at positions that can be regarded as being the same withinan allowable margin delta in a small region (I′+dX, J′+dY) centered at aposition (I′, J′) in the coordinate system of the first image, that is,in a rectangular region whose corners are at (I′−dX, J′−dY), (I′−dX,J′+dY), (I′+dX, J′−dY), and (I′+dX, J′+dY), wherein dX and dY are valuesthat define the region in the vicinity of the position (x, y) and theyare set to be equal to, for example, 2 times the size of a pixel in theX direction and Y direction, respectively.

In other words, the calculation circuit 24 determines (k′, ig, jg) and(k″, ib, jb) that satisfy formula (38) within the small region (I′+dX,J′+dY) centered at the position (I′, J′) of interest in the coordinatesystem of the first image, that is, in the rectangular region whosecorners are at (I′−dX, J′−dY), (I′−dX, J′+dY), (I′+dX, J′−dY), and(I′+dX, J′+dY).

Tthe calculation circuit 24 then acquires pixel values Gobs(k′, ig, jg)and Bobs(k′″, ib, jb) respectively corresponding to (k′, ig, jg) and(k′″, ib, jb) determined.

The calculation circuit 24 determines (k′, ig, jg) and (k′″, ib, jb)that satisfy formula (38) for all possible combinations of k′ and k′″within the range form 1 to N.

In general, a plurality of combinations of (k′, ig, jg) and (k′″, ib,jb) is detected, and the calculation circuit 24 plots pixel valuesGobs(k′, ig, jg) and Bobs(k′″, ib, jb) corresponding to the detected(k′, ig, jg) and (k′″, ib, jb) in the GB space in which the horizontalaxis represents the G signal (Gobs(k′, ig, jg)) and the vertical axisrepresents the B signal (Bobs(k′″, ib, jb)).

Thus, in the seventh embodiment, in addition to spring constraints givenby equations (25) to (27) in the third embodiment, the correlationbetween green and blue colors shown is added as a constrain.

The calculation circuit 24 performs principal component analysis for theplurality of points plotted in the GB space, wherein each plotted pointcorresponds to a pair of pixel values Gobs(k′, ig, jg) and Bobs(k′″, ib,jb) that the calculation circuit 24 has detected as satisfying thedistance constraint (38).

The calculation circuit 24 then determines a principal direction as aresult of the principal component analysis and determines a lineQg′×G+Qb′×R=Qc′ representing the determined direction (axis), where Qg′,Qb′, and Qc′ are constants associated with the line in the GB space, andG and B are variables associated with the G and B signals in the GBspace. Furthermore, the calculation circuit 24 determines the variancein a direction perpendicular to the principal direction.

Herein, a spring with a natural length of 0 and with a spring constant(strength) H is introduced. The spring constant may be an arbitrarymonotonically decreasing function of the variance of the componentperpendicular to the direction of the principal component determined inthe GB space. For example, H=(1/variance) may be used. In any case, thestrength of the spring (spring constant) increases with decreasingvariance.

One end of the spring with the spring constant H is connected to point(Lg(I′, J′), Lb(I′, J′)) in the GB space, and the other end of thespring is connected such that it can movable to an arbitrary point onthe line Qg′×G+Qb′×B=Qc′. As a result, the point (Lg(I′, J′), Lb(I′,J′)) is pulled in a direction of the perpendicular line from the point(Lg(I′, J′), Lb(I′, J′)) to the line Qg′×G+Qb′×B =Qc′. That is, thepoint (Lg(I′, J′), Lb(I′, J′)) is pulled such that the color correlationconstraint (between the G and B signals) is satisfied.

Herein, if the distance between the point (Lg(I′, J′), Lb(I′, J′)) andthe line Qgl×G+Qb′×B=Qcl extending in the direction of the maincomponent is represented by u, then the pulling force of the spring withthe spring constant H exerted on the point (Lg(I′, J′), Lb(I′, J′)) isgiven by H×u (hereinafter, this force will be referred to as GB springforce). The GB spring force H×u in the GB space is broken down into a Gsignal component and a B signal component, and these components arerespectively added to the left-hand sides of equations (25) and (27)such that the color correlation is taken into account in equations (25)and (27).

Although in the present embodiment, the constraint associated with thecolor correlation in the GR space and the constraint associated with thecolor correlation in the GB space, a similar constraint associated witha color correlation between the R signal and the B signal (in a RBspace) may also be employed in addition to the above constraints.

Now, referring to a flow chart shown in FIG. 39, a correct imageestimation process in step S4 shown in FIG. 2 according to a seventhembodiment of the present invention is described.

Steps S171 to S177 are similar to steps S71 to S77 in the correct imageestimation process according to the third embodiment described abovewith reference to FIG. 24.

That is, in step S171, the calculation circuit 24 selects a position(I′, J′) in the reference coordinate system (hereinafter, such aselected position will be referred to a position (I′, J′) of interest).Note that the position (I′, J′) of interest is at the center (i−0.5,j−0.5) of a (i-th, j-th) pixel of the first image taken as the referenceimage.

Thereafter, the process proceeds from step S171 to step S172. In stepS172, the calculation circuit 24 determines all combinations of integers(k, ig, jg) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (θk, Tkx, Tky, Sk), thecenter position (ig−0.5, jg−0.5) of the pixel that senses a greencomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS173.

In step S173, the calculation circuit 24 produces a spring equation inthe form (25) for green light intensity, using all combinations of (k,ig, jg) determined in step S172. Thereafter, the process proceeds tostep S174.

In step S174, the calculation circuit 24 determines all combinations ofintegers (k, ir, jr) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ir−0.5, jr−0.5) of the pixel that senses a redcomponent of the k-th image, for the position (I′, J′) of interest ofeach of first to N-th images. Thereafter, the process proceeds to stepS175.

In step S175, the calculation circuit 24 produces a spring equation inthe form (26) for red light intensity, using all combinations of (k, ir,jr) determined in step S174. Thereafter, the process proceeds to stepS176.

In step S176, the calculation circuit 24 determines all combinations ofintegers (k, ib, jb) that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (θk, Tkx, Tky,Sk), the center position (ib−0.5, jb−0.5) of the pixel that senses ablue component of the k-th image, for the position (I′, J′) of interestof each of first to N-th images. Thereafter, the process proceeds tostep S177.

In step S177, the calculation circuit 24 produces a spring equation inthe form (27) for blue light intensity, using all combinations of (k,ib, jb) determined in step S176. Thereafter, the process proceeds tostep S178.

In step S178, the calculation circuit 24 determines all pairs of (k′,ig, jg) and (k″, ir, jr) that satisfy formula (37) in the region definedby (I′+dx, J′+dY) and centered at the position (I′, J′) of interest.Furthermore, the calculation circuit 24 plots, in the GR space, points(Gobs(k′, ig, jg) and Robs(k″, ir, jr)) identified by the determinedpairs of (k′, ig, jg) and (k″, ir, jr), and performs principal componentanalysis. The calculation circuit 24 then determines the variance in adirection perpendicular to the principal direction and employs thereciprocal of the variance as the spring constant H. Furthermore, thecalculation circuit 24 determines a line Qg×G+Qr×R=Qc representing thedirection of the principal component and defines an unknown quantity H×uindicating the GR spring force where u is the distance between a point(Lg(I′, J′), Lr(I′, J′)) in the GR space and the line Qg×G+Qr×R=Qc. Thecalculation circuit 24 adds the G signal component of the GR springforce H×u to the left-hand side of equation (25) and adds the R signalcomponent of the GR spring force H×u to the left-hand side of equation(26). Thereafter, the process proceeds from step S178 to step S179.

In step S179, and the calculation circuit 24 determines all pairs of(k′, ig, jg) and (k′″, ib, jb) that satisfy formula (38) in the regiondefined by (I′±dx, J′±dY) and centered at the position (I′, J′) ofinterest. Furthermore, the calculation circuit 24 plots, in the GBspace, points (Gobs(k′, ig, jg) and Bobs(k′″, ib, jb)) identified by thedetermined pairs of (k′, ig, jg) and (k′″, ib, jb), and performsprincipal component analysis. The calculation circuit 24 then determinesthe variance in a direction perpendicular to the principal direction andemploys the reciprocal of the variance as the spring constant H.Furthermore, the calculation circuit 24 determines a lineQg′×G+Qb′×R=Qc′ representing the direction of the principal componentand defines an unknown quantity H×u indicating the GB spring force whereu is the distance between a point (Lg(I′, J′), Lb(I′, J′)) in the GBspace and the line Qg′×G+Qb′×R=Qc′. The calculation circuit 24 adds theG signal component of the GB spring force H×u to the left-hand side ofequation (25) and adds the B signal component of the GB spring force H×uto the left-hand side of equation (27). Thereafter, the process proceedsfrom step S179 to step S180.

In step S180, the calculation circuit 24 solves linear equations, thatis, the spring equation associated with the green light intensity andincluding the additional terms associated with the color correlations inthe GR and GB spaces, the spring equation associated with the red lightintensity and including the additional term associated with the colorcorrelation in the GR space, and the spring equation associated with theblue light intensity and including the additional term associated withthe color correlation in the GB space thereby determining the true greenlight intensity Lg(I′, J′), the true red light intensity Lr(I′, J′), andthe true blue light intensity Lb(I′, J′) at the position (I′, J′) ofinterest. Thereafter, the process proceeds to step S181.

In step S181, the calculation circuit 24 determines whether theabove-described process is completed for all possible positions (I′,J′), that is, whether the true green light intensity Lg(I′, J′), thetrue red light intensity Lr(I′, JY), and the true blue light intensityLb(I′, J′) have been determined by taking the center position of each ofall pixels of the first image as the position (I′, J′) of interest.

If it is determined in step S181 that the process is not completed forall possible positions (I′, J′) of interest, the process returns to stepS171 to repeat steps S171 to S181. That is, the calculation circuit 24selects, as a new position of interest, a position (I′, J′) that has notyet been selected as the position of interest, and determines the truegreen light intensity Lg(I′, J′), the true red light intensity Lr(I′,J′), and the true blue light intensity Lb(I′, J′) at the position (I′,J′) of interest.

On the other hand in the case in which it is determined in step S181that the process is completed by taking all possible position (I′, J′)as the position of interest, the process proceeds to step S182. In stepS182, the calculation circuit 24 estimates a correct image (signal) fromthe true green light intensity Lg(I′, J′), the true red light intensityLr(I′, J′), and the true blue light intensity Lb(I′, J′) determined instep S180. The resultant image is supplied as an output image to the D/Aconverter 9 or the codec 12, and the current process is exited. Forexample, for the (i-th, j-th) pixel, the calculation circuit 24estimates the correct image signal thereof from the true green lightintensity Lg(i−0.5, j−0.5) determined as the green value (G signal) instep S180, the true red light intensity Lr(i−0.5, j−0.5) determined asthe red value (R signal) in step S180, and the true blue light intensityLb(i−0.5, j−0.5) determined as the blue value (B signal) in step S180.The calculation circuit 24 estimates the output image by estimating theimage signal for all pixels whose center is located at the selectedposition (I′, J′).

As described above, in the seventh embodiment, the error due to noise orthe like is reduced by taking into account the color correlation whenthe output image is produced. This makes it possible to produce asharper output image more likely to be the correct image.

To provide a better understanding of the features of the presentinvention, the difference between the present invention and theconventional technique is further described below.

FIG. 40 shows a processing flow of the method according to the presentinvention and that of the conventional method.

As shown in FIG. 40, many methods (denoted by (A) in FIG. 40) to convertan image output from a single-plane sensor (for example, a Bayer-arrayimage in which each pixel has only one of R, G, and B signals) to anormal color image (having R, G, and B signals for each pixel) are knownin the art. In fact, this conversion capability is implemented in somedigital still cameras using a single-plane sensor available in themarket. This conversion is called demosaicing.

A method (denoted by (B) in FIG. 40) of producing a sharp color image(having R, G, and B signals for each pixel) (with a high signal-to-noiseratio) from a plurality of dark color images (having R, G, and B signalsfor each pixel) (with a low signal-to-noise ratio) is also known in theart. Specific examples of this method include Japanese Unexamined PatentApplication Publication No. 2000-217032 and Japanese Unexamined PatentApplication Publication No. 2000-224460 cited earlier in Background Art.

By combining these two methods, it is possible to produce a sharp colorimage (having R, G, and B signals for each pixel) (with a highsignal-to-noise ratio) from an image output from a single-plane sensor(for example, a Bayer-array image in which each pixel has only one of R,G, and B signals).

More specifically, a demosaicing process 202-m denoted by (A) in FIG. 40is performed on each of a plurality (m) of dark images 201-m output froma single-plane sensor (for example, Bayer-array images in which eachpixel has only one of R, G, and B signals) (with a low signal-to-noiseratio) to produce dark color images (having R, G, and B signals for eachpixel) (with a low signal-to-noise ratio) 203-m. Thereafter, a sharpcolor image (having R, G, and B signals for each pixel) (with a highsignal-to-noise ratio) 204 is produced from the color images 203-m bythe method denoted by (B) in FIG. 40 such as a method disclosed inJapanese Unexamined Patent Application Publication No. 2000-217032 orJapanese Unexamined Patent Application Publication No. 2000-224460. Notethat m is an integer indicating the number of images.

Interpolation or similar processing performed in the demosacing processcauses a reduction in sharpness of the image. In the process disclosedin Japanese Unexamined Patent Application Publication No. 2000-217032 or2000-224460, interpolation is performed for registration among digitallysampled data, and thus degradation in sharpness also occurs. That is,interpolation is performed in the process denoted by (A) in FIG. 40 andalso in the process (B). The interpolation performed a total of twotimes causes significant degradation in sharpness. The interpolation isused to reproduce missing data at a particular point from data atadjacent points. The more times interpolation is performed, the moredegradation in sharpness occurs.

In contrast, in the present invention, a sharp color image (having R, G,and B signals for each pixel) (with a high signal-to-noise ratio) isproduced directly from an image output from a single-plane sensor (forexample, a Bayer-array image in which each pixel has only one of R, G,and B signals). Therefore, interpolation or similar processing isperformed only once. Thus, it is possible to produce a sharper imagethan the conventional method can. As can be seen from the abovediscussion, it is obvious that the present invention is different fromthe conventional technique and the present invention has greatadvantages.

Another problem with the conventional technique is that demosacing isperformed for each image. For example, when eight images are given asthe plurality of images, demosacing (denoted by (A) in FIG. 40) isperformed eight times. This implies that a large amount of computationis required.

The different between the present invention and the conventionaltechnique is further described below.

In the conventional technique disclosed in Japanese Unexamined PatentApplication Publication No. 2000-217032 cited earlier in Background Art,when a plurality of images are superimposed, images are displaced toregister them such that a positional displacement among original imagesdue to a camera shake is cancelled out. To produce images displaced byan amount necessary to cancel out the camera shake, interpolation isnecessary. Superimposing N images together is performed by superimposingN data at each pixel of an output image. After N images aresuperimposed, the pixel value of each pixel is divided by a particularvalue (equal to N or equal to (optimum exposure time)/(actual exposuretime)/N). It is also known in the art to achieve a high resolution imageby capturing a plurality of images while displacing pixel positions by afixed amount each time one image is captured and superimposing theplurality of captured images while properly registering them. In thistechnique, the amount of displacement is fixed, the pixel value of eachpixel of the output image is divided (normalized) by a fixed value.

In contrast, in the first or second embodiment according to the presentinvention, the number of pixel values observed at a given position (x,y) of an output image available to estimate a true value variesdepending on a situation. For example, when a picture of a scene istaken by a user, there is a possibility that there are a large number ofinput pixel values Gobs(k, ig, jg) in the vicinity of a position (x, y).In such a case, a large number of constraints associated with theposition (x, y) in the form of a mathematical expression (20) areavailable. Conversely, if there are only a small number of input pixelsGobs(k, ig, jg) in the vicinity of the position (x, y), only a smallnumber of constraints associated with the position (x, y) in the form ofthe mathematical expression (20) are available. As described above, thenumber of mathematical constraints associated with each position (x, y)of an output image is variable depending on the state in which a pictureis taken (more particularly, for example, depending on the camerashake). That is, in the process of estimating the true value at eachpixel, observation points of candidates for pixel values to be used inthe estimation and the number thereof are adaptively changed dependingon the state in which the picture is taken (depending on the motionamong the plurality of captured images). For example, the number ofobservation points of candidates for pixel values, and a particularnumber of observations are selected from the candidates. Alternatively,both the number of observation points of candidates and the number ofobservation points to be selected may be changed. The above-describedtwo methods may be selectively used depending on the positions, numbers,and/or distribution of observation points.

In the third to seventh embodiments of the present invention, whenimages are superimposed, images are displaced so as to cancel outdisplacement due to a camera shake. However, in the Bayer-arraystructure, there is not necessarily data to be added, at all outputpositions. When a k-th image is displaced in position, if there is noposition-corrected data within a 2×2 pixel area shown in FIG. 21, anydata of the k-th image has no contribution to those 2×2 pixels of theoutput image. Conversely, if there are two position-corrected datawithin the 2×2 pixel area shown in FIG. 21, the two (weighted) data areadded to the output pixel. For example, if ten data are added at aparticular pixel, the final pixel value of the output image is obtainedby dividing the sum by ten. If 3.5 data are added together at anotherpixel, the pixel value at that pixel of the output image is obtained bydividing the sum by 3.5. That is, in the present invention, the numberof times summation E is executed in equations (25), (26), and (27)varies depending on the state in which a picture is taken (moreparticularly, for example, depending on the camera shake). That is, inthe process of estimating the true value at each pixel, observationpoints of candidates for pixel values to be used in the estimation andthe number thereof are adaptively changed depending on the state inwhich the picture is taken (depending on the motion among the pluralityof captured images).

In the present invention, as described above, when a sharp image isrestored from a plurality of images output from a single-plane sensor(Bayer-array sensor), the number of constraint equations or the weightsare varied depending on the situation. In this regard, the presentinvention is very different from the conventional techniques.

The sequence of processing steps described above may be performed bymeans of hardware or software. For example, the digital camera 1 may berealized by executing a program on a computer such as that shown in FIG.41.

In FIG. 41, a CPU (Central Processing Unit) 301 performs variousprocesses in accordance with a program stored in a ROM (Read OnlyMemory) 302 or in accordance with a program loaded into a RAM (RandomAccess Memory) 303 from a storage unit 308. The RAM 303 is also used tostore data used by the CPU 301 in the execution of various processes.The CPU 301 executes the process performed by the motion detector 23 andthe calculation circuit 24 of the signal processor 7.

The CPU 301, the ROM 302, and the RAM 303 are connected to each othervia a bus 304. The bus 304 is also connected to an input/outputinterface 305.

The input/output interface 305 is connected to an input unit 306including a keyboard, a mouse, and/or the like, an output unit 307including a display such as a CRT (Cathode Ray Tube) or a LCD (LiquidCrystal Display), a loudspeaker, and/or the like, the storage unit 308such as a hard disk, and a communication unit 309 including a modem, aterminal adapter, and/or the like. The communication unit 309 allowscommunication via a network such as the Internet. The imaging unit 311includes the imaging device 4 shown in FIG. 1 and serves to captureimages of a subject. The resultant image data of the subject is suppliedto the CPU 301 via the input/output interface 305.

The input/output interface 305 is also connected to a drive 310, asrequired. A storage medium such as a magnetic disk 321, an optical disk322, a magnetooptical disk 323, or a semiconductor memory 324 is mountedon the drive 310 as required, and a computer program is read from thestorage medium and installed into the storage unit 308, as required.

Now, a correct image estimation process performed by the signalprocessor 7 according to an eighth embodiment of the present inventionis described. In this eighth embodiment, the third embodiment describedabove with reference to FIG. 24 is partially improved.

In the third embodiment described above, data (light intensity) sensedby each pixel of the imaging device 4 (FIG. 1) is regarded aspoint-sampled data, the relationship between a pixel value observed atthe center of each pixel and an ideal image signal of a sharp imageincluding no blur due to a camera shake is represented by the springmodel, and a sharp image likely to be a true image is determined basedon the spring model.

For example, of G, R, and B signals of an image signal of a sharp imagewith no blur due to camera shake, estimation of the true G signalaccording to the third embodiment is performed as follows. That is, theconstraint on the true green light intensity Lg(I′, J′) to be determinedfor the center position (I′, J′) of each pixel is expressed by thespring equation (25) associated with the green light intensity, asdescribed earlier.

In equation (25), {√2−F((x, y), (I′, J′))} of the spring constant{√2−F((x, y), (I′, J′))}/(Mk×E) represents the weight that decreaseswith distance between the position (I′, J′) and the position (x, y) inthe vicinity of the position (I′, J′). This means that the pulling forceof the spring exerted between the pixel value Gobs(k, ig, jg) at theposition (x, y) and the true green light intensity Lg(I′, J′) at theposition (I′, J′) decreases with distance between the position (I′, J′)and the position (x, y). Conversely, the pulling force of the springincreases with decreasing distance between the position (I′, J′) and theposition (x, y).

On the other hand, (Mk×E) of the spring constant {√2−F((x, y), (I′,J′))}/(Mk×E) in equation (25) represents residual noise E included thatcannot be removed by the correlated double sampler 5 (FIG. 1) gained upby a factor of Mk (k=1 to N). Thus, 1/(Mk×E) represents a weightassociated with noise, which decreases with noise E. Therefore, thepulling force of the spring exerted between the pixel value Gobs(k, ig,jg) at the position (x, y) and the true green light intensity Lg(I′, J′)at the position (I′, J′) decreases with increasing noise E and increaseswith decreasing noise.

In the eighth embodiment, {√2−F((x, y), (I′, J′))} in equation (25),that is, the weight depending on the distance between the position (I′,J′) and the position (x, y) of the pixel value Gobs(k, ig, jg)(hereinafter, referred to simply as a distance weight) is replaced witha function including cubic functions: Cubic(I′−x)×Cubic(J′−y). That is,in the eighth embodiment, {√2−F((x, y), (I′, J′))} in equation (25) isreplaced with Cubic(I′−x)×Cubic(J′−y).

The cubic function Cubic(z) is in a form (39) shown below.$\begin{matrix}{{{Cubic}(z)} = \left\{ \begin{matrix}{{\left( {a + 2} \right){z}^{3}} - {\left( {a + 3} \right){z}^{2}} + {1\quad\left( {{z} < 1} \right)}} \\{{a{z}^{3}} - {5a{z}^{2}} + {8a{z}} - {4a\quad\left( {1 \leq {z} < 2} \right)}} \\{0\quad\left( {2 \leq {z}} \right)}\end{matrix} \right.} & (39)\end{matrix}$where a is a predetermined constant, which is set, for example, to −1.

As shown in FIG. 42, the cubic function Cubic(z) has a value of 0 when2≦|z| or when |z|=1, and the cubic function Cubic(z) is negative when1<|z|<2. When |z|<1, Cubic(z) is positive and decreases with |z|. Inother words, the cubic function Cubic(z) serves as a lowpass filter inthe frequency domain (for Fourier-transformed variable).

In equation (25), if p-th pixel value Gobs(k, ig, jg) subjected tosummation Σ is denoted by vp, the spring constant {√2−F((x, y), (I′,J′))}/(Mk×E) associated with the p-th pixel value vp is denoted as aspring constant Kp, and if the resultant equation is solved with respectto the true green light intensity Lg(I′, J′), equation (40) is obtained.$\begin{matrix}{{{Lg}\left( {I^{\prime},J^{\prime}} \right)} = \frac{\sum{K_{p}v_{p}}}{\sum K_{p}}} & (40)\end{matrix}$where Σ denotes the sum with respect to p.

From equation (40), it can be seen that, in the spring model, the truegreen light intensity Lg(I′, J′) is given by the weighted sum of pixelvalues v_(p) wherein the spring constants Kp are employed as weights.The true red light intensity Lr(I′, J′) and the true blue lightintensity Lb(I′, J′) can be determined in a similar manner.

In FIG. 43, for the position (I′, J′), when positions (ig−0.5, jg−0.5)of N captured images are converted into positions (x, y) in thereference coordinate system (coordinate system defined on the firstimage), all combinations of integers k, ig, and jg that satisfyI′−2≦x<I′+2 and J′−2≦y<J′+2 are determined, and points G11 to G19 atwhich pixel values Gobs(k, ig, jg) are observed are shown.

In FIG. 43, for a position (I′, J′) denoted by a solid circle,combinations of integers (k, ig, jg) identifying nine pixel valuesGobs(k, ig, jg) observed at points G11 to G19 denoted by open circlesare obtained. Note that combinations of integers k, ig and jg aredifferent among points Gl1 to G19.

In the third embodiment described earlier, as shown in FIG. 21, for aposition (I′, J′), positions (ig−0.5, jg−0.5) on N captured images areconverted into positions (x, y) in the reference coordinate system(coordinate system defined on the first image), and all combinations ofintegers k, ig, and jg that satisfy I′−1≦x<I′+1 and J′−1≦y<J′+1 aredetermined. In contrast, in the eighth embodiment, as shown in FIG. 43,combinations of integers k, ig, and jg that satisfy I′−2≦x<I′+2 andJ′−2≦y<J′+2 are determined. As described above, the cubic functionCubic(z) in the form (39) has a value depending on argument z for −2<z<2(Cubic(z) has a value of 0 for any z in the range of |z|>2), and thetrue green light intensity Lg(I′, J′) at the position (I′, J′) isestimated using pixel values Gobs(k, ig, jg) observed in the range−2≦z<2, that is, in the range defined by I′−2≦x<I′+2 and J′−2≦y<J′+2. Inthe third embodiment described earlier, a position (ig−0.5, jg−0.5) onthe k-th captured image is converted to a position ((ig−0.5)_((k)),(ig−0.5)_((k))) in the reference coordinate system defined on the firstcaptured image as follows. That is, the motion detector 23-(k−1)determines the affine conversion parameters (θk, Tkx, Tky, Sk) includinga rotation angle θk, a scale Sk, and a parallel displacement (Tkx, Tky)from the positional relationship between the first captured image andthe k-th captured image. The resultant conversion parameters (θk, Tkx,Tky, Sk) are supplied to the calculation circuit 24. Based on theconversion parameters (θk, Tkx, Tky, Sk) from the motion detector23-(k−1), the calculation circuit 24 converts the position (ig−0.5,jg−0.5) on the k-th captured image to a position ((ig−0.5)_((k)),(ig−0.5)_((k))) in the reference coordinate system in accordance withaffine conversion equation (3).

On the other hand, in the eighth embodiment, the motion detector23-(k−1) determines affine conversion parameters of an affine conversionequation (41) representing the positional relationship between the firstcaptured image and the k-th captured image. $\begin{matrix}{\begin{pmatrix}{X1}_{(k)} \\{Y1}_{(k)}\end{pmatrix} = {{\begin{pmatrix}{ak}^{\prime} & {bk}^{\prime} \\{ck}^{\prime} & {dk}^{\prime}\end{pmatrix}\begin{pmatrix}{Xk} \\{Yk}\end{pmatrix}} + \begin{pmatrix}{Tkx}^{\prime} \\{Tky}^{\prime}\end{pmatrix}}} & (41)\end{matrix}$

More specifically, the motion detector 23-(k−1) determines a matrix(ak′, bk′, ck′, dk′) and a 2-dimensional vector (Tkx′, Tky′) in equation(41) representing the positional relationship between the first capturedimage and the k-th captured image, and supplies the resultant matrix(ak′, bk′, ck′, dk′) and the 2-dimensional vector (Tkx′, Tky′) to thecalculation circuit 24. Hereinafter, the affine conversion parametersincluding the matrix (ak′, bk′, ck′, dk′) and the 2-dimensional vector(Tkx′, Tky′) in equation (41) will be referred to simply as conversionparameters (ak′, bk′, dk′, Tkx′, Tky′).

If respective parameters are given such that ak′=dk′=Sk×cos(θk) and−bk′=ck′=Sk×sin(θk), equation (41) becomes equivalent to equaion (3).

In equation (41), as in equaion (3), (Xk, Yk) represents the position ofa pixel of the k-th captured image, and (X1 _((k)), Y1 _((k)))represents a position on the first captured image (in the referencecoordinate system) obtained by affine-converting the position (Xk, Yk)in accordance with equation (41). Note that a subscript (k) is used toindicate a position of the k-th captured image converted into thereference coordinate system. If the conversion parameters for the firstcaptured image are given as (a1′, b1′, c1′, d1′, T1x′, T1y′)=(1, 0, 0,1, 0, 0), then equation (41) holds for all values of k in the range from1 to N.

For example, the motion detector 23-1 determines affine conversionparameters (a2′, b2′, c2′, d2′, T2x′, T2y′) of equation (42)representing the positional relationship between the first capturedimage and a second captured image, and supplies the resultant conversionparameters (a2′, b2′, c2′, d2′, T2x′, T2y′) to the calculation circuit24. $\begin{matrix}{\begin{pmatrix}{X1}_{(2)} \\{Y1}_{(2)}\end{pmatrix} = {{\begin{pmatrix}{a2}^{\prime} & {b2}^{\prime} \\{c2}^{\prime} & {d2}^{\prime}\end{pmatrix}\begin{pmatrix}{X2} \\{Y2}\end{pmatrix}} + \begin{pmatrix}{T2x}^{\prime} \\{T2y}^{\prime}\end{pmatrix}}} & (42)\end{matrix}$

On the other hand, the motion detector 23-2 determines affine conversionparameters (a3′, b3′, c3′, d3′, T3x′, T3y′)of equation (43) representingthe positional relationship between the first captured image and a thirdcaptured image, and supplies the resultant conversion parameters (a3′,b3′, c3′, d3′, T3x′, T3y′) to the calculation circuit 24.$\begin{matrix}{\begin{pmatrix}{X1}_{(3)} \\{Y1}_{(3)}\end{pmatrix} = {{\begin{pmatrix}{a3}^{\prime} & {b3}^{\prime} \\{c3}^{\prime} & {d3}^{\prime}\end{pmatrix}\begin{pmatrix}{X3} \\{Y3}\end{pmatrix}} + \begin{pmatrix}{T3x}^{\prime} \\{T3y}^{\prime}\end{pmatrix}}} & (43)\end{matrix}$

Equation (43) represents that a position (X3, Y3) on the third capturedimage is displaced, because of camera shake or the like, by an amountcorresponding to the conversion parameters (a3′, b3′, c3′, d3′, T3x′,T3y′) with respect to a position (X1, Y1) on the first captured image.

Instead of determining the conversion parameters (ak′, bk′, ck′, dk′,Tkx′, Tky′) from the positional relationship of the k-th captured imagewith respect to the first captured image in the above-described manner,the conversion parameters (ak′, bk′, ck′, dk′, Tkx′, Tky′) may bemechanically determined from a signal output from an acceleration sensordisposed on the digital camera 1. In the case of camera shaking thatoccurs when a camera is held in hands, the positional relationshipbetween the first captured image and the k-th captured image includessubstantially no rotation component. Therefore, the rotation componentin the conversion parameters (ak′, bkt, ck′, dk′, Tkx′, Tky′) may beneglected (that is, it may be assumed that no rotation occurs).

In the eighth embodiment, the weight associated with noise in equation(25) is set based on the assumption that all N captured images aregained up by the same factor (by means of n′-bit shifting). That is, inthe eight embodiment, it is assumed that all N images (input images)captured by the imaging device 4 have brightness that is 1/M=1/Mk timesthe brightness obtained with optimum exposure. For example, if capturedimages have brightness equal to ⅛ times the brightness of an imagecaptured with optimum exposure, then M=8, and thus shifter 21 (FIG. 1)gains up the captured image by performing 3-bit shifting.

In the eighth embodiment, because the shifter 21 gains up all N capturedimages by the same factor, the weight 1/(Mk×E) associated with noise inequation (25) can be simply rewritten as 1/(M×E).

Thus, in this eighth embodiment, equation (25) used in the thirdembodiment can be rewritten as equation (44). $\begin{matrix}{{\sum\left\lbrack {\frac{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)}}{M \times E} \times \left\{ {{{Gobs}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0} & (44)\end{matrix}$

For the given position (I′, J′), the sum E in equation (44) is taken forall combinations (k, ig, jg) that satisfy I′−2≦x<I′+2 and J′−2≦y<J′+2with respect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (ak′, bk′, ck′,dk′, Tkx′, Tky′), the position (ig−0.5, jg−0.5). In the specific exampleshown in FIG. 43, the sum is taken for nine combinations of integers (k,ig, jg) corresponding to points G11 to G19.

Equation (44) can be regarded as a spring equation associated with thegreen light intensity using the cubic function Cubic(z) according to theeighth embodiment, corresponding to the spring equation (25) associatedwith the green light intensity.

In equation (44), the weight 1/(M×E) associated with noise is constantregardless of the combinations of integers (k, ig, jg) subjected to thesummation Y, and thus the weight 1/(M×E) can be moved out to the outsideof Σ. Thus, equation (44) can be rewritten as (45). $\begin{matrix}{{{\frac{1}{M \times E}{\sum\left\lbrack {{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)} \times \left\{ {{{Gobs}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack}} = 0}{{\sum\left\lbrack {{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)} \times \left\{ {{{Gobs}\left( {k,{ig},{jg}} \right)} - {{Lg}\left( {I^{\prime},J^{\prime}} \right)}} \right\}} \right\rbrack} = 0}} & (45)\end{matrix}$

If equation (45) is solved with respect to the true green lightintensity Lg(I′, J′), equation (46) is obtained.∑Cubic(I^(′) − x) × Cubic(J^(′) − y) × Gobs(k, ig, jg) = ∑Cubic(I^(′) − x) × Cubic(J^(′) − y) × Lg(I^(′), J^(′))${{Lg}\left( {I^{\prime},J^{\prime}} \right)} = \frac{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)} \times {{Gobs}\left( {k,{ig},{jg}} \right)}}}{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)}}}$

In this eighth embodiment, the true green light intensity Lg(I′, J′) isdetermined using equation (46).

Equation (46) indicates that the green light intensity Lg(I′, J′) can beobtained by calculating the weighted sum of pixel values Gobs(k, ig, jg)by using Cubic(I′−x)×Cubic(J′−y) as the weight. Note that the weightCubic(I′−x)×Cubic(J′−y) serves as a lowpass filter for the distancebetween the position (I′, J′) and the position (x, y) of the pixel valueGobs(k, ig, jg).

Hereinafter, equation (46) will be referred to as a weighted sumequation associated with the green light intensity. The denominator andthe numerator of equation (46) associated with the weighted sum of greenlight intensities are separately rewritten as equations (47) and (48),respectively.ΣCubic(I′−x)×Cubic(J′−y)×Gobs(k, ig, ig)  (47)ΣCubic(I′−x)×Cubic(J−y)  (48)

If the original data (light intensity) in the state in which the pixelvalue Gobs(k, ig, jg) is not yet gained up by a factor of M by theshifter 21 is denoted as DGobs(k, ig, ig), then equation (46) can bewritten as (49). $\begin{matrix}{{{Lg}\left( {I^{\prime},J^{\prime}} \right)} = \frac{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)} \times M \times D_{{Gobs}{({k,{ig},{jg}})}}}}{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)}}}} & (49)\end{matrix}$

The true red light intensity Lr(I′, J′) and the true blue lightintensity Lb(I′, J′) at the position (I′, J′) of interest can berespectively represented by equations (50) and (51), which is similar toequation (46). $\begin{matrix}{{{Lr}\left( {I^{\prime},J^{\prime}} \right)} = \frac{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)} \times {{Robs}\left( {k,{ir},{jr}} \right)}}}{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)}}}} & (50)\end{matrix}$

Hereinafter, equation (50) will be referred to as a weighted sumequation associated with the red light intensity. For the given position(I′, J′), the sum E in equation (50) is taken for all combinations (k,ir, jr) that satisfy I′−2≦x<I′+2 and J′−2≦y<J′+2 with respect to theposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (ak′, bk′, ck′, dk′, Tkx′,Tky′), the position (ir−0.5, jr−0.5). $\begin{matrix}{{{Lb}\left( {I^{\prime},J^{\prime}} \right)} = \frac{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)} \times {{Bobs}\left( {k,{ib},{jb}} \right)}}}{\sum{{{Cubic}\left( {I^{\prime} - x} \right)} \times {{Cubic}\left( {J^{\prime} - y} \right)}}}} & (51)\end{matrix}$

Hereinafter, equation (51) will be referred to as a weighted sumequation associated with the blue light intensity. For the givenposition (I′, J′), the sum E in equation (51) is taken for allcombinations (k, ib, jb) that satisfy I′−2≦x<I′+2 and J′−2≦y<J′+2 withrespect to the position (x, y) in the reference coordinate systemobtained by converting, based on conversion parameters (ak′, bk″, ck′,dk′, Tkx′, Tky′), the position (ib−0.5, jb−0.5).

The denominator and the numerator of equation (50) associated with theweighted sum of green light intensities are separately rewritten asequations (52) and (53), respectively.ΣCubic(I′−x)×Cubic(J′−y)×Robs(k, ir,ir)  (52)ΣCubic(I′−x)×Cubic(J′−y)  (53)

The denominator and the numerator of equation (51) associated with theweighted sum of green light intensities are separately rewritten asequations (54) and (55), respectively.ΣCubic(I′−x)×Cubic(J′−y)×Bobs(k, ib, jb).ΣCubic(I′−x)×Cubic(J′−y)  (55)

By calculating the weighted sum associated with the green lightintensity according to equation (46), the weighted sum associated withthe red light intensity according to equation (50), and the weighted sumassociated with the blue light intensity according to equation (51), thecalculation circuit 24 determines the true green light intensity Lg(I′,J′), the true red light intensity Lr(I′, J′), and the true blue lightintensity Lb(I′, J′). The process performed by the calculation circuit24 to determine those light intensities is referred to as a normalprocess to distinguish it from an exceptional process that will bedescribed later.

The weighted sum associated with the green light intensity given byequation (46), the weighted sum associated with the red light intensitygiven by equation (50), and the weighted sum associated with the bluelight intensity given by equation (51) will be further discussed below.

Equation (46) indicates that the weighted sum associated with the greenlight intensity is obtained by dividing the sum of products of the pixelvalue Gobs(k, ig, jg) at the position (x, y) and the distance weightCubic(I′−x)×Cubic(J′−y) (that is, the sum of pixel values Gobs(k, ig,jg) weighted with the distance weight Cubic(I′−x)×Cubic(J′−y)) by thesum of the distance weights Cubic(I′−x)×Cubic(J′−y), that is, bydividing the value given by equation (47) that is equal to the numeratorof equation (46) by the value given by equation (48) that is equal todenominator of equation (46).

If the value given by equation (48) that is equal to the denominator ofequation (46) becomes equal to 0, equation (46) representing the truegreen light intensity Lg(I′, J′) becomes unstable (uncertain), and thusthe result is unreliable. In other words, at a position (I′, J′) atwhich the value given by equation (48) that is equal to the denominatorof equation (46) becomes equal to 0, small noise (error) included in thepixel values Gobs(k, ig, jg) in equation (46) that is equal to thenumerator of equation (48) is divided by the denominator equal to 0, andthus the result becomes very large. In this case, the true green lightintensity Lg(I′, J′) calculated according to equation (46) includeslarge noise and is unreliable.

The value given by equation (48) that is equal to the denominator ofequation (46) becomes equal to 0, for example, when at least one ofcubic functions Cubic(I′−x) and Cubic(J′−y) becomes equal to 0 over allthe range of the summation in equation (48). As can be seen from FIG.42, cubic functions Cubic(I′−x) or Cubic(J′−y) becomes equal to 0 whenI′−x=±1 or J′−y=±1, that is, when x=I′±1 or y=J′±1.

Therefore, for a given position (I′, J′), when positions (ig−0.5,jg−0.5) are converted into the reference coordinate system usingconversion parameters (ak′, bk′, ck′, dk′, Tkx′, Tky′), if all positions(x, y) of pixel values Gobs(k, ig, jg) of (k, ig, jg) appearing withinthe range defined by I′−2≦x<I′+2 and J′−2≦y<J′+2 satisfy x=I′±1 ory=J′±1, the denominator of equation (46) representing the true greenlight intensity Lg(I′, J′) becomes equal to 0 (or nearly equal to 0),and thus the result becomes unstable and unreliable. However, allpositions (x, y) of pixel values Gobs(k, ig, jg) appearing within therange defined by I′−2≦x<I′+2 and J′−2≦y<J′+2 satisfy x=I′±1 or y=J′±1only in a very rare and exceptional state. Hereinafter, such a specialstate will be referred to simply as an exceptional state.

Hereinafter, a position (x, y) converted from a position (k, i, j) on ak-th captured image into the reference coordinate system usingconversion parameters (ak′, bk′, ck′, dk′, Tkx′, Tky′) will be referredsimply as a converted (corrected) position (x, y).

FIG. 44 shows an exceptional state at a position (I′, J′).

In the specific example shown in FIG. 44, when positions (ig−0.5,jg−0.5) are converted using conversion parameters (ak′, bk′, ck′, dk′,Tkx′, Tky′), converted points (converted positions) G11′ and G15′ arelocated at positions where x=I′−1 is satisfied, and converted pointsG12′ and G16′ are located at positions where x=I′+1 is satisfied.

Furthermore, converted points G13′ and G14′ are located at positionswhere y=J′−1 is satisfied, and converted points G17′ and G19′ arelocated at positions where y=J′+1 is satisfied.

In the specific example shown in FIG. 44, converted positions (x, y)falling within the region in the coordinate system defined byI′−2≦x<I′+2 and J′−2≦y<J′+2 have a special relation with the position(I′, J′), that is, x=I′±1 or y=J′±1 for all combinations of integers (k,ig, jg). In this state, there is no G signal data (pixel value Gobs(k,ig, jg)) in a region (I′±1, J′+1) (a square region with a size of 2×2)centered at the position (I′, J′).

In such a state (exceptional state), if the true green light intensityLg(I′, J′) at the position (I′, J′) is determined in accordance withequation (46), the resultant true green light intensity Lg(I′, J′) isunreliable (unstable), as described above.

To avoid the above problem, when a pixel at the position (I′, J′) is insuch an exceptional state, the calculation circuit 24 performs anexceptional process.

More specifically, for a given position (I′, J′) of interest, when thetrue green light intensity Lg(I′, J′) of a pixel of an output image atthe position (I′, J′) of interest is estimated (hereinafter, such apixel will be referred to simply as a pixel of interest), if the pixelof the interest (the position (I′, J′) of the pixel of interest) is inthe exceptional state, the calculation circuit 24 determines the truegreen light intensity Lg(I′, J′) at the position (I′, J′) of interest ofthe pixel of the interest in the exceptional state by using pixel valuesGobs(k, ig, jg) of the k-th captured image whose corrected position (x,y) is located in the vicinity of the position of a pixel of the outputimage located in the vicinity of the pixel of interest, in addition topixel values Gobs(k, ig, jg) of the k-th captured image whose correctedposition (x, y) is located in the vicinity of the position (I′, J′) ofinterest. As for the pixel (adjacent pixel) located adjacent to thepixel of interest located at the position (I′, J′) of interest, pixelsat positions (I′−1, J′), (I′+1, J′), (I′, J′−1), and (I′, J′+1) shown inFIG. 45 may be employed.

As described earlier with reference to FIG. 3, the imaging device 4 ofthe digital camera 1 according to the present embodiment has aBayer-array structure. In the Bayer-array structure, pixels for sensingthe green light component are disposed every two pixel positions in bothX and Y direction (FIG. 3). In this eighth embodiment, it is assumedthat the pixels of the imaging device 4 arranged such that W pixels arearranged in a horizontal (X) direction and H pixels are arranged in avertical (Y) direction, and thus the imaging device 4 has a total of W xH pixels.

When there is no observed value (pixel value) Gobs(k, ig, jg) of the Gsignal in the vicinity of the position (I′, J′) of interest, the pixelof interest at the position (I′, J′) of interest cannot be a pixel forsensing the green light component in the Bayer-array structure.

For example, as shown in FIG. 46, when the pixel of interest at theposition (I′, J′) of interest is a pixel B12 denoted by an open circlefor sensing the blue light component in the Bayer-array structure, thereis a pixel for sensing the green light component at an immediatelyadjacent location to the left or right of or above or below the pixelB12. Similarly, when the pixel of interest is a pixel of the imagingdevice 4 for sensing the red or blue light component, there is a pixelfor sensing the green light component at an immediately adjacentlocation to the left or right of or above or below the pixel ofinterest.

Therefore, when there is no observed value Gobs(k, ig, jg) of the Gsignal in the vicinity of the position (I′, J′) of interest (when thereis no green pixel of the captured image whose converted position (x, y)is located in the vicinity of the position (I′, J′) of interest), thereis an observed value of the G signal in the vicinity of one of positions(I′, J′−1), (I′, J′+1), (I′−1, J′), and (I′+1, J′) of adjacent pixels atimmediately adjacent locations above or below or to the left or right ofthe pixel of interest at the position (I′, J′) of interest. When thereis an observed value Gobs(k, ig, jg) of the G signal in the vicinity ofparticular one of adjacent pixels located at positions (I′−1, J′),(I′+1, J′), (I′, J′−1), and (I′, J′+1), the particular one of adjacentpixels is not in the exceptional state. For this particular adjacentpixel located at position (I′−1, J′), (I′+1, J′), (I′, J′−1), or (I′,J′+1), a reliable value of the green light intensity Lg(I′, J′) can bedetermined using equation (46).

As described above, when the pixel of interest is in the exceptionalstate, the calculation circuit 24 determines the true green lightintensity Lg(I′, J′) at the position (I′, J′) of interest by using pixelvalues Gobs(k, ig, jg) of pixels whose corrected position (x, y) islocated in the vicinity of an adjacent pixel located at (I′−1, J′),(I′+1, J′), (I′, J′−1), or (I′, J′+1) adjacent to the pixel of interest,in addition to pixel values Gobs(k, ig, jg) of pixels whose correctedposition (x, y) is located in the vicinity of the position (I′, J′) ofinterest.

More particularly, in this case, the calculation circuit 24 determinesthe true green light intensity Lg(I′, J′) at the position (I′, J′) ofinterest by performing the exceptional process in accordance withequation (56). $\begin{matrix}{{{Lg}\left( {I^{\prime},J^{\prime}} \right)} = {\left\{ {{{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},J^{\prime}} \right)} + {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},J^{\prime}} \right)} + {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},J^{\prime}} \right)} + {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} - 1}} \right)} + {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} + 1}} \right)}} \right\}/\left\{ {{{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},J^{\prime}} \right)} + {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},J^{\prime}} \right)} + {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},J^{\prime}} \right)} + {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} - 1}} \right)} + {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} + 1}} \right)}} \right\}}} & (56)\end{matrix}$

Equation (56) represents that the true green light intensity Lg(I′, J′)is obtained by dividing the sum of the values of the numerator ofequation (46) obtained in the normal process for total of fivepositions, that is, the position (I′, J′) of interest of the pixel ofinterest and the positions (I′−1, J′), (I′+1, J′), (I′, J′−1), and (I′,J′+1) of pixels adjacent to the pixel of interest by the sum of thevalues of the denominator of equation (46) obtained in the normalprocess for total of five positions, that is, the position (I′, J′) ofinterest of the pixel of interest and the positions I′−1, J′), (I′+1,J′), (I′, J′−1), and (I′, J′+1) of pixels adjacent to the pixel ofinterest, that is, by dividing the sum of the values of equation (47) bythe sum of the values of equation (48).

Note that when the pixel of interest is in the exceptional state, theequation used to determine the true green light intensity Lg(I′, J′) atthe position (I′, J′) of interest is not limited to equation (56) butother equations may be used. For example, equation (57) shown below maybe used for this purpose. $\begin{matrix}{{{LG}\left( {I^{\prime},J} \right)} = {\left\{ {{\left( {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},J^{\prime}} \right)} \right)/\left( {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},J^{\prime}} \right)} \right)} + {\left( {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},J^{\prime}} \right)} \right)/\left( {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},J^{\prime}} \right)} \right)} + {\left( {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} - 1}} \right)} \right)/\left( {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} - 1}} \right)} \right)} + {\left( {{numerator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} + 1}} \right)} \right)/\left( {{denominator}\quad{value}\quad{of}\quad G\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} + 1}} \right)} \right)}} \right\}/4}} & (57)\end{matrix}$

Equation (57) represent that the estimated true green light intensityLg(I′, J′) at the (I′, J′) of interest is obtained by calculating themean value of true green light intensities Lg(Il−1, J′), Lg(I′+1, J′),Lg(I′, J′−1) and Lg(I′, J′+1) calculated using equation (46) for fourpositions (I′−1, J′), (I′+1, J′), (I′, J′−1), and (I′, J′+1) of pixelsadjacent to the pixel of interest.

The weighted sum associated with the red light intensity given byequation (50) will be further discussed below.

As with the case in which the true green light intensity Lg(I′, J′) iscalculated according to equation (46), when the true red light intensityLr(I′, J′) is calculated according to equation (50), the solutionbecomes unstable if the pixel of interest is in the exceptional state.More specifically, for a given position (I′, J′) of interest, whenpositions (ir−0.5, jr−0.5) are converted into the reference coordinatesystem using conversion parameters (ak′, bk′, ck′, dk′, Tkx′, Tky′),there is a possibility that all converted positions (x, y) of pixelvalues Robs(k, ir, jr) appearing within the range defined by I′−2≦x<I′+2and J′−2≦y<J′+2 satisfy x=I′±1 or y=J′±1 as shown in FIG. 44. In such astate (exceptional state), there is no R signal data (pixel valueRobs(k, ir, jr)) in a region (I′±1, J′±1) (a square region with a sizeof 2×2) centered at the position (I′, J′).

In this case, the calculation circuit 24 performs an exceptional processas described below.

That is, the calculation circuit 24 determines the true red lightintensity Lr(I′, J′) at the position (I′, J′) of interest of the pixelof the interest in the exceptional state by using pixel values Robs(k,ir, jr) of the k-th captured image whose corrected position is locatedin the vicinity of the position of a pixel of the output image locatedin the vicinity of the pixel of interest, in addition to pixel valuesGobs(k, ir, jr) of the k-th captured image whose corrected position islocated in the vicinity of the pixel of interest. As for the pixel(adjacent pixel) located adjacent to the pixel of interest located atthe position (I′, J′) of interest, pixels at positions I′−1, J′−1), (I′,J′−1), (I′+1, J′−1), (I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1),and (I′+1, J′+1) shown in FIG. 47 may be employed.

When there is no observed value (pixel value) Robs(k, ir, jr) of the Rsignal in the vicinity of the position (I′, J′) of interest, the pixelof interest at the position (I′, J′) of interest cannot be a pixel forsensing the red light component in the Bayer-array structure.

For example, as shown in FIG. 48, when the pixel of interest at theposition (I′, J′) of interest is a pixel G11 denoted by an open circlefor sensing the green light component in the Bayer-array structure,there is a pixel for sensing the red light component at an immediatelyadjacent location above or below the pixel G11.

On the other hand, for example, as shown in FIG. 48, when the pixel ofinterest at the position (I′, J′) of interest is a pixel G22 denoted byan open circle for sensing the green light component in the Bayer-arraystructure, there is a pixel for sensing the red light component at animmediately adjacent location to the left or right of the pixel G22.

Furthermore, for example, as shown in FIG. 48, when the pixel ofinterest at the position (I′, J′) of interest is a pixel B14 denoted byan open circle for sensing the blue light component in the Bayer-arraystructure, there is a pixel for sensing the red light component at animmediately adjacent location to the upper right, lower right, upperleft, or lower left of the pixel B14.

Similarly, when the pixel of interest is a pixel other than pixels forsensing the red light component, there is a pixel for sensing the redlight component at an immediately adjacent location to the left, right,upper right, lower right, upper left, lower left of the pixel B14 orabove or below the pixel B14.

Therefore, when there is no observed value Robs(k, ir, jr) of the Rsignal in the vicinity of the position (I′, J′) of interest (when thereis no red pixel of the captured image whose converted position (x, y) islocated in the vicinity of the position (I′, J′) of interest), there isan observed value of the R signal in the vicinity of one of positions(I′−1, J′−1), (I′, J′−1), (I′+1, J′−1), (I′−1, J′), (I′+1, J′), (I′−1,J′+1), (I′, J′+1), and (I′+1, J′+1) of adjacent pixels at immediatelyadjacent locations above or below or to the left, right, upper right,lower right, upper left, lower left of the pixel of interest at theposition (I′, J′) of interest. When there is an observed value Robs(k,ir, jr) of the R signal in the vicinity of particular one of adjacentpixels located at positions (I′−1, J′−1), (I′, J′−1), (I′+1, J′−1),(I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1), and (I′+1, J′+1), theparticular one of adjacent pixels is not in the exceptional state. Forthis particular adjacent pixel located at position (I′−1, J′−1), (I′,J′−1), (I′+1, J′−1), (I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1),or (I′+1, J′+1), a reliable value of the red light intensity Lr(I′, J′)can be determined using equation (50).

As described above, when the pixel of interest is in the exceptionalstate, the calculation circuit 24 determines the true red lightintensity Lr(I′, J′) at the position (I′, J′) of interest by using pixelvalues Robs(k, ir, jr) of pixels whose corrected position (x, y) islocated in the vicinity of an adjacent pixel located at (I′−1, J′−1),(I′, J′−1), (I′+1, J′−1), (I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′,J′+1), or (I′+1, J′+1) adjacent to the pixel of interest, in addition topixel values Robs(k, ir, jr) of pixels whose corrected position (x, y)is located in the vicinity of the position (I′, J′) of interest.

More particularly, in this case, the calculation circuit 24 determinesthe true red light intensity Lr(I′, J′) at the position (I′, J′) ofinterest by performing the exceptional process in accordance withequation (58). $\begin{matrix}{{{Lr}\left( {I^{\prime},J^{\prime}} \right)} = {\left( {{{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {I^{\prime},J^{\prime}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},{J^{\prime} - 1}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} - 1}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},{J^{\prime} - 1}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},J^{\prime}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},J^{\prime}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},{J^{\prime} + 1}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} + 1}} \right)} + {{numerator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},{J^{\prime} + 1}} \right)}} \right)/\left( \left( {{{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {I^{\prime},J^{\prime}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},{J^{\prime} - 1}} \right)} + \left( {{{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} - 1}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},{J^{\prime} - 1}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},J^{\prime}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},J^{\prime}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} - 1},{J^{\prime} + 1}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {I^{\prime},{J^{\prime} + 1}} \right)} + {{denominator}\quad{value}\quad{of}\quad R\quad{signal}\quad{at}\quad\left( {{I^{\prime} + 1},{J^{\prime} + 1}} \right)}} \right)} \right. \right.}} & (58)\end{matrix}$

Equation (58) represents that the true red light intensity Lr(I′, J′) isobtained by dividing the sum of the values of the numerator of equation(50) obtained in the normal process for total of nine positions, thatis, the position (I′, J′) of interest of the pixel of interest and thepositions (I′−1, J′−1), (I′, J′−1), (I′+1, J′−1), (I′−1, J′), (I′+1,J′), (I′−1, J′+1), (I′, J′+1), and (I′+1, J′+1) of pixels adjacent tothe pixel of interest by the sum of the values of the denominator ofequation (50) obtained in the normal process for total of ninepositions, that is, the position (I′, J′) of interest of the pixel ofinterest and the positions (I′−1, J′−1), (I′, J′−1), (I′+1, J′−1),(I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1), and (I′+1, J′+1) ofpixels adjacent to the pixel of interest, that is, by dividing the sumof the values of equation (52) by the sum of the values of equation(53).

As with equation (57) associated with the G signal, instead of usingequation (58), the true red light intensity Lr(I′, J′) at the position(I′, J′) of interest may be determined by calculating the mean value oftrue red light intensities Lr(I′−1, J′−1), Lr(I′, J′−1), Lr(I′+1, J′−1),Lr(I′−1, J′), Lr(I′+1, J′), Lr(I′−1, J′+1), Lr(I′, J′+1), and Lr(I′+1,J′+1) of pixels at eight positions (I′−1, J′−1), (I′, J′−1), (I′+1,J′−1), (I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1), and (I′+1,J′+1) adjacent to pixel of interest.

When the true blue light intensity Lb(I′, J′) is determined using theweighted sum equation (51) associated with the blue light intensity, aswith the true green light intensity Lg(I′, Ji) determined using theweighted sum equation (46) associated with the green light intensity oras with the true red light intensity Lr(I′, J′) determined using theweighted sum equation (50) associated with the red light intensity, thesolution becomes unstable if the pixel of interest at the position ofinterest is in the exceptional state.

In this case, the calculation circuit 24 performs an exceptional processas described below.

In the Bayer-array structure, pixels for sensing the blue lightcomponent are arranged in a similar manner in terms of positionalrelationship to the manner in which pixels for sensing the red lightcomponent are arranged. Therefore, the calculation circuit 24 determinesthe true blue light intensity Lb(I′, J′) at the position (I′, J′) ofinterest of the pixel of the interest in the exceptional state accordingto equation (59) similar to equation (58).Lb(I^(′), J^(′)) = (numerator  value  of  B  signal  at  (I^(′), J^(′)) + numerator  value  of  B  signal  at  (I^(′) − 1, J^(′) − 1) + numerator  value  of  B  signal  at  (I^(′), J^(′) − 1) + numerator  value  of  B  signal  at  (I^(′) + 1, J^(′) − 1) + numerator  value  of  B  signal  at  (I^(′) − 1, J^(′)) + numerator  value  of  B  signal  at  (I^(′) + 1, J^(′)) + numerator  value  of  B  signal  at  (I^(′) − 1, J^(′) + 1) + numerator  value  of  B  signal  at  (I^(′), J^(′) + 1) + numerator  value  of  B  signal  at  (I^(′) + 1, J^(′) + 1))/((denominator  value  of  B  signal  at  (I^(′), J^(′)) + denominator  value  of  B  signal  at  (I^(′) − 1, J^(′) − 1) + (denominator  value  of  B  signal  at  (I^(′), J^(′) − 1) + denominator  value  of  B  signal  at  (I^(′) + 1, J^(′) − 1) + denominator  value  of  B  signal  at  (I^(′) − 1, J^(′)) + denominator  value  of  B  signal  at  (I^(′) + 1, J^(′)) + denominator  value  of  B  signal  at  (I^(′) − 1, J^(′) + 1) + denominator  value  of  B  signal  at  (I^(′), J^(′) + 1) + denominator  value  of  B  signal  at  (I^(′) + 1, J^(′) + 1))

Equation (59) represents that the true blue light intensity Lb(I′, J′)is obtained by dividing the sum of the values of the numerator ofequation (51) obtained in the normal process for total of ninepositions, that is, the position (I′, J′) of interest of the pixel ofinterest and the positions (I′−1, J′−1), (I′, J′−1), (I′+1, J′−1), (I−1,il), (I′+1, J′), (I′−1, J′+1), (I′, J′+1), and (I′+1, J′+1) of pixelsadjacent to the pixel of interest the sum of the values of thedenominator of equation (51) obtained in the normal process for total ofnine positions, that is, the position (I′, J′) of interest of the pixelof interest and the positions (I′−1, J′−1), (I′, J′−1), (I′+1, J′−1),(I−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1), and (I′+1, J′+1) ofpixels adjacent to the pixel of interest, that is, by dividing the sumof the values of equation (54) by the sum of the values of equation(55).

As with equation (57) associated with the G signal, instead of usingequation (59), the true blue light intensity Lb(I′, J′) at the position(I′, J′) of interest may be determined by calculating the mean value oftrue blue light intensities Lb(I′−1, J′−1), Lb(I′, J′−1), Lb(I′+1,J′−1), Lb(I′−1, J′), Lb(I′+1, J′), Lb (I−1, J′+1), Lb(I′, J′+1), andLb(I′+1, J′+1) of pixels at eight positions (I′−1, J′−1), (I′, J′−1),(I′+1, J′−1), (I′−1, J′), (I′+1, J′), (I′−1, J′+1), (I′, J′+1), and(I′+1, J′+1) adjacent to pixel of interest.

Now, referring to flow charts shown in FIGS. 49 and 50, a correct imageestimation process in step S4 shown in FIG. 2 according to an eighthembodiment of the present invention is described.

First, in step S201, the calculation circuit 24 sets a variable J′ of(I′, J′) to 1 to indicate the position in the Y direction of a pixel inthe reference coordinate system. Thereafter, the process proceeds tostep S202.

In step S202, the calculation circuit 24 calculates sets a variable I′of (I′, J′) to 1 to indicate the position in the X direction of a pixelin the reference coordinate system. Thereafter, the process proceeds tostep S203. Note that variables I′ and J′ are also used to indicate theposition in the X and Y directions of a pixel of an output image.

In step S203, the calculation circuit 24 selects the position (I′, J′)as a position of interest, and, for the position (I′, J′) of interest,the calculation circuit 24 determines all combinations of integers (k,ig, jg) that satisfy I′−2≦x<I′+2 and J′−2≦y<J′+2 with respect to theconverted position (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (ak′, bk′, ck′, dk′, Tkx′,Tky′), the center position (ig−0.5, jg−0.5) of the pixel that senses thegreen component of the k-th image, for all of first to N-th images.Thereafter, the process proceeds to step S204.

In step S204, the calculation circuit 24 calculates the denominator ofthe weighted sum equation associated with the green light intensityaccording to equation (48) and the numerator of the weighted sumequation associated with the green light intensity according to equation(47) by using all combinations of integers (k, ig, jg) determined instep S203. The calculation circuit 24 stores the result of thecalculation in a memory (not shown). Thereafter, the process proceeds tostep S205.

In step S205, for the position (I′, J′) of interest, the calculationcircuit 24 determines all combinations of integers (k, ir, jr) thatsatisfy I′−2≦x<I′+2 and J′−2≦y<J′+2 with respect to the convertedposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (ak′, bk′, ck′, dk′, Tkx′,Tky′), the center position (ir−0.5, jr−0.5) of the pixel that senses thered component of the k-th image, for all of first to N-th images.Thereafter, the process proceeds to step S206.

In step S206, the calculation circuit 24 calculates the denominator ofthe weighted sum equation associated with the red light intensityaccording to equation (53) and the numerator of the weighted sumequation associated with the red light intensity according to equation(52) by using all combinations of integers (k, ir, jr) determined instep S205. The calculation circuit 24 stores the result of thecalculation in the memory. Thereafter, the process proceeds to stepS207.

In step S207, for the position (I′, J′) of interest, the calculationcircuit 24 determines all combinations of integers (k, ib, jb) thatsatisfy I′−2≦x<I′+2 and J′−2≦y<J′+2 with respect to the convertedposition (x, y) in the reference coordinate system obtained byconverting, based on conversion parameters (ak′, bk′, ck′, dk′, Tkx′,Tky′), the center position (ib−0.5, jb−0.5) of the pixel that senses theblue component of the k-th image, for all of first to N-th images.

In step S208, the calculation circuit 24 calculates the denominator ofthe weighted sum equation associated with the blue light intensityaccording to equation (55) and the numerator of the weighted sumequation associated with the blue light intensity according to equation(54) by using all combinations of integers (k, ib, jb) determined instep S207. The calculation circuit 24 stores the result of thecalculation in the memory. Thereafter, the process proceeds to stepS209.

In step S209, the calculation circuit 24 determines whether the variableI′ is equal to the number of pixels W in the X direction. If it isdetermined in step S209 that the variable I′ is not equal to the numberof pixel W, that is, if steps S203 to S208 are not completed for allpixels whose Y position is J′, the process proceeds to step S210. Instep S210, the calculation circuit 24 increments the variable I′ by 1.Thereafter, the process returns to step S203.

In the case in which it is determined in step S209 that the variable I′is equal to the number of pixel W, that is, if steps S203 to S208 arecompleted for all pixels whose Y position is J′, the process proceeds tostep S211.

In step S211, the calculation circuit 24 determines whether the variableJ′ is equal to the number of pixels H in the Y direction. If it isdetermined in step S211 that the variable J′ is not equal to the numberof pixel H, that is, if steps S203 to S208 are not completed for all Ypositions of the imaging device 4, the process proceeds to step S212. Instep S212, the calculation circuit 24 increments the variable J′ by 1.Thereafter, the process returns to step S202.

On the other hand, if it is determined in step S211 that the variable J′is equal to the number of pixel H, that is, if steps S203 to S208 arecompleted for all Y positions of the imaging device 4, the processproceeds to step S213 in FIG. 50. Note that the numbers of pixels W andH are equal to the numbers of pixels of the output image in the X and Ydirections, respectively.

In step S213, as in step S201, the calculation circuit 24 sets thevariable J′ to 1. Thereafter, the process proceeds to step S214.

In step S214, ass in step S202, the calculation circuit 24 sets thevariable I′ to 1. Thereafter, the process proceeds to step S215.

In step S215, the calculation circuit 24 calculates employs the position(I′, J′) as the position of interest, and calculates the true greenlight intensity Lg(I′, J′) at the position (I′, J′) of interest.Thereafter, the process proceeds to step S216. More specifically, instep S215 described above, the true green light intensity Lg(I′, J′) atthe position (I′, J′) of interest is determined by performing either thenormal process (first calculation process) using the weighted sumequation (46) associated with the green light intensity or theexceptional process (second calculation process) according to equation(56), as will be described in further detail later.

In step S216, the calculation circuit 24 calculates employs the position(I′, J′) as the position of interest, and calculates the true red lightintensity Lr(I′, J′) at the position (I′, J′) of interest. Thereafter,the process proceeds to step S217. More specifically, in step S216described above, the true red light intensity Lr(I′, J′) at the position(I′, J′) of interest is determined by performing either the normalprocess (first calculation process) using the weighted sum equation (50)associated with the red light intensity or the exceptional process(second calculation process) according to equation (58), as will bedescribed in further detail later.

In step S217, the calculation circuit 24 employs the position (I′, J′)as the position of interest, and calculates the true blue lightintensity Lb(I′, J′) at the position (I′, J′) of interest. Thereafter,the process proceeds to step S218. More specifically, in step S217described above, the true blue light intensity Lb(I′, J′) at theposition (I′, J′) of interest is determined by performing either thenormal process (first calculation process) using the weighted sumequation (51) associated with the blue light intensity or theexceptional process (second calculation process) according to equation(59), as will be described in further detail later.

In step S218, the calculation circuit 24 determines whether the variableI′ is equal to the number W of pixels in the X direction. If it isdetermined in step S218 that the variable I′ is not equal to the numberof pixel W, that is, if steps S215 to S217 are not completed for allpixels whose Y position is J′, the process proceeds to step S219. Instep S219, the calculation circuit 24 increments the variable I′ by 1.Thereafter, the process returns to step S215.

In the case in which it is determined in step S218 that the variable I′is equal to the number of pixel W, that is, if steps S215 to S217 arecompleted for all pixels whose Y position is J′, the process proceeds tostep S220.

In step S220, the calculation circuit 24 determines whether the variableJ′ is equal to the number of pixels H in the Y direction. If it isdetermined in step S220 that the variable J′ is not equal to the numberof pixel H, that is, if steps S215 to S217 are not completed for all Ypositions of the imaging device 4, the process proceeds to step S221. Instep S221, the calculation circuit 24 increments the variable J′ by 1.Thereafter, the process returns to step S214.

On the other hand, if it is determined in step S220 that the variable J′is equal to the number of pixel H, that is, if steps S215 to S217 arecompleted for all Y positions of the imaging device 4, the processproceeds to step S222.

In step S222, the calculation circuit 24 estimates an output image(signal) from the true green light intensity Lg(I′, J′), the true redlight intensity Lr(I′, J′), and the true blue light intensity Lb(I′, J′)determined respectively steps S215, S216, and S217. The resultant outputimage is supplied to the D/A converter 9 or the codec 12. Thereafter,the processing flow returns from the current process. More specifically,for the (i-th, j-th) pixel, the calculation circuit 24 estimates acorrect image signal of the (i-th, j-th) pixel from the true green lightintensity Lg (=Lg(I′, J′)) determined as the green value (G signal) instep S215, the true red light intensity Lr(i−0.5, j−0.5) (=Lr(I′, J′))determined as the red value (R signal) in step S216, and the true bluelight intensity Lb(i−0.5, j−0.5) (=Lb(I′, J′)) determined as the bluevalue (B signal) in step S217. The calculation circuit 24 estimates anoutput image by performing the above-described process for all pixels ofthe output image whose center is located at the position (I′, J′)(=(i−0.5, j−0.5)).

Now referring to a flow chart shown in FIG. 51, the process performed instep S215 in FIG. 50 to calculate the true green light intensity Lg(I′,J′) at the position (I′, J′) of interest is described below.

First, in step S251, the calculation circuit 24 determines whether theabsolute value of the denominator of the weighted sum of green lightintensities in equation (46) calculated in step S204 in FIG. 49 for theposition (I′, J′) of interest, that is, the absolute value of equation(48), is equal to or greater than a threshold value. The threshold valueis used to determine whether the absolute value of equation (48) shouldbe regarded as being equal to 0 and thus the exception handling processshould be performed. The threshold value is set in advance in thecalculation circuit 24. The threshold value may be set variably to avalue specified by a user.

If it is determined in step S251 that the absolute value of equation(48) for the position (I′, J′) of interest is equal to or greater thanthe threshold value, that is, if the absolute value of equation (48) forthe position (I′, J′) of interest is not so small that it can beregarded as equal to 0, the process proceeds to step S252. In step S252,the calculation circuit 24 selects the normal process and calculates theweighted sum equation (46) associated with the green light intensity.That is, the calculation circuit 24 divides the value of the numeratorof the weighted sum equation (46) associated with the green lightintensity calculated in step S204, that is, the value of equation (47)by the value of the denominator of the weighted sum equation (46)associated with the green light intensity calculated in step S204, thatis, the value of equation (48). Thus, in step S252, the true green lightintensity Lg(I′, J′) at the position (I′, J′) of interest is determined.

On the other hand, if it is determined in step S251 that the absolutevalue of equation (48) for the position (I′, J′) of interest is lessthan the threshold value, that is, the absolute value of equation (48)is equal to or nearly equal to 0, the process proceeds to step S253. Instep S253, the calculation circuit 24 selects the exceptional processand performs it. That is, the calculation circuit 24 determines thelight intensity Lg(I′, J′) of the true green color at a position (I′,J′) of interest by calculating equation (56).

Now referring to a flow chart shown in FIG. 52, the process performed instep S216 in FIG. 50 to calculate the true red light intensity Lr(I′,J′) at the position (I′, J′) of interest is described below.

First, in step S271, and the calculation circuit 24 determines whetherthe absolute value of the denominator of the weighted sum of red lightintensities in equation (50) calculated in step S206 in FIG. 49 for theposition (I′, J′) of interest, that is, the absolute value of equation(53), is equal to or greater than a threshold value. The threshold valueis used to determine whether the absolute value of equation (53) shouldbe regarded as being equal to 0 and thus the exception handling processshould be performed. The threshold value is set in advance in thecalculation circuit 24. The threshold value may be set variably to avalue specified by a user. Note that the threshold value may or may notbe equal to the threshold value used in step S251 in FIG. 51.

If it is determined in step S271 that the absolute value of equation(53) for the position (I′, J′) of interest is equal to or greater thanthe threshold value, that is, if the absolute value of equation (53) forthe position (I′, J′) of interest is not so small that it can beregarded as equal to 0, the process proceeds to step S272. In step S272,the calculation circuit 24 selects the normal process and calculates theweighted sum equation (50) associated with the red light intensity. Thatis, the calculation circuit 24 divides the value of the numerator of theweighted sum equation (50) associated with the red light intensitycalculated in step S206, that is, the value of equation (52) by thevalue of the denominator of the weighted sum equation (50) associatedwith the red light intensity calculated in step S206, that is, the valueof equation (53). Thus, in step S272, the true red light intensityLr(I′, J′) at the position (I′, J′) of interest is determined.

On the other hand, if it is determined in step S271 that the absolutevalue of equation (53) for the position (I′, J′) of interest is lessthan the threshold value, that is, the absolute value of equation (53)is equal to or nearly equal to 0, the process proceeds to step S273. Instep S273, the calculation circuit 24 selects the exceptional processand performs it. That is, the calculation circuit 24 determines thelight intensity Lr(I′, J′) of the true red color at a position (I′, J′)of interest by calculating equation (58).

Now referring to a flow chart shown in FIG. 53, the process performed instep S217 in FIG. 50 to calculate the true blue light intensity Lg(I′,J′) at the position (I′, J′) of interest is described below.

First, in step S291, and the calculation circuit 24 determines whetherthe absolute value of the denominator of the weighted sum of blue lightintensities in equation (49) calculated in step S208 in FIG. 49 for theposition (I′, J′) of interest, that is, the absolute value of equation(55), is equal to or greater than a threshold value. The threshold valueis used to determine whether the absolute value of equation (55) shouldbe regarded as being equal to 0 and thus the exception handling processshould be performed. The threshold value is set in advance in thecalculation circuit 24. The threshold value may be set variably to avalue specified by a user. Note that the threshold value may or may notbe equal to the threshold value used in step S251 in FIG. 51 or thatused in step S271 in FIG. 52.

If it is determined in step S291 that the absolute value of equation(55) for the position (I′, J′) of interest is equal to or greater thanthe threshold value, that is, if the absolute value of equation (55) forthe position (I′, J′) of interest is not so small that it can beregarded as equal to 0, the process proceeds to step S292. In step S292,the calculation circuit 24 selects the normal process and calculates theweighted sum equation (51) associated with the blue light intensity.That is, the calculation circuit 24 divides the value of the numeratorof the weighted sum equation (51) associated with the blue lightintensity calculated in step S208, that is, the value of equation (54)by the value of the denominator of the weighted sum equation (51)associated with the blue light intensity calculated in step S208, thatis, the value of equation (55). Thus, in step S292, the true blue lightintensity Lb(I′, J′) at the position (I′, J′) of interest is determined.

On the other hand, if it is determined in step S291 that the absolutevalue of equation (55) for the position (I′, J′) of interest is lessthan the threshold value, that is, the absolute value of equation (55)is equal to or nearly equal to 0, the process proceeds to step S293. Instep S293, the calculation circuit 24 selects the exceptional processand performs it. That is, the calculation circuit 24 determines thelight intensity Lb(I′, J′) of the true blue color at a position (I′, J′)of interest by calculating equation (59).

As described above, in the eighth embodiment, the true green lightintensity Lg(I′, J′), the true red light intensity Lr(I′, J′), and thetrue blue light intensity Lb(I′, J′) are determined by calculating theweighted sum using the cubic function having a characteristic similar tothat of the lowpass filter as the weight depending on the distancebetween the position (I′, J′) of interest and the converted position (x,y) in the vicinity of the position (I′, J′) of interest, therebydetermining a sharp image very close to an ideal image.

In the eighth embodiment, for the position (I′, J′) of interest, whenpositions (ig−0.5, jg−0.5) are converted into the reference coordinatesystem using conversion parameters (ak′, bk′, ck′, dk′, Tkx′, Tky′), ifone or more converted positions (x, y) of pixel values Gobs(k, ig, jg)fall within the range defined by I′−2≦x<I′+2 and J′−2≦y<J′+2, then theweighted sum of those pixel values Gobs(k, ig, jg) whose convertedpositions (x, y) are located in the above-described range in thevicinity of the position (I′, J′) of interest is calculated inaccordance with the weighted sum equation (46) associated with the greenlight intensity, and the result is employed as the true green lightintensity Lg(I′, J′) (normal process).

However, if the absolute value of the denominator of the weighted sumequation (46) associated with the green light intensity calculated forthe position (I′, J′) of interest, that is, the value of equation (48),is less than the predetermined small threshold value that can beregarded as substantially equal to 0, the solution of the weighted sumequation (46) becomes unstable. In such a case, the weighted sum ofpixel values Gobs(k, ig, jg) whose conversion position (x, y) falls inthe vicinity of the position (I′, J′) of interest and pixel valuesGobs(k, ig, jg) whose conversion position (x, y) falls in the vicinityof a pixel adjacent to the pixel of interest is calculated in accordancewith the weighted sum equation (56), and the result is employed as thetrue green light intensity Lg(l′, Ji) (exceptional process).

The true red light intensity Lr(I′, J′) and the true blue lightintensity Lb(I′, J′) are determined in a similar manner.

Thus, it is possible to obtain a good output image includingsubstantially no noise.

The normal process and the exceptional process will be further discussedbelow from another point of view.

For example, for the green light intensity, in the normal processdescribed above, the weighted sum is calculated using pixel valuesGobs(k, ig, jg) of pixels of the captured image whose converted position(x, y) falls within the vicinity of the position (I′, J′) of interest ofthe pixel of interest. On the other hand, in the exceptional process,the weighted sum is calculated using pixel values Gobs(k, ig, jg) ofpixels of the captured image whose converted position (x, y) fallswithin the vicinity of the position (I′, J′) of interest of the pixel ofinterest and also using pixel values Gobs(k, ig, jg) of pixels of thecaptured image whose converted position (x, y) falls within the vicinityof the position of a pixel adjacent to the pixel of interest.

That is, in the exceptional process, the weighted sum is calculatedusing not only pixel values Gobs(k, ig, jg) of pixels of the capturedimage whose converted position (x, y) falls within the vicinity of theposition (I′, J′) of the pixel of interest but also pixel values Gobs(k,ig, jg) of pixels of the captured image whose converted position (x, y)falls within the vicinity of a pixel adjacent to the pixel of interest.

That is, as described above, in the normal process, the true green lightintensity Lg(I′, J′) at the position (I′, J′) of interest is determinedby calculating the weighted sum of pixel values Gobs(k, ig, jg) of thecaptured image observed within the region in the vicinity of theposition (I′, J′) of interest, that is, within the region defined byI′−2≦x<I′+2 and J′−2≦y<J′+2. In contrast, in the exceptional process,the true green light intensity Lg(I′, J′) at the position (I′, J′) ofinterest is determined by calculating the weighted sum of pixel valuesGobs(k, ig, jg) of the captured image observed within the region in thevicinity of the position (I′, J′) of interest, which is greater than thevicinity region used in the normal process, that is, more specificallywithin the region defined by I′−3≦x<I′+3 and J′−3≦y<J′+3.

In other words, when the true green light intensity Lg(I′, J′) at theposition (I′, J′) of interest is determined, the region in the vicinityof the position (I′, J′) of interest is defined by I′−3≦x<I′+3 and J′−3≦y<J′+3. In the normal process, the true green light intensity Lg(I′,J′) at the position (I′, J′) of interest is determined by calculatingthe weighted sum equation (46), in which the weight is set to 0 forthose pixel values Gobs(k, ig, jg) that are observed within the vicinityregion defined above but that are outside the region defined by I′−2<x<I′+2 and J′−2≦y<J′+2. On the other hand, in the exceptional process,the true green light intensity Lg(I′, J′) at the position (I′, J′) ofinterest is determined according to equation (56), in which, for thosepixel values Gobs(k, ig, jg) that are observed within the vicinityregion defined above but that are outside the region defined byI′−2≦x<I′+2 and J′−2≦y<J′+2, that is, for pixel values Gobs(k, ig, jg)observed in the vicinity of a pixel adjacent to the pixel of interest,the weight is set not to o but to a value given by the cubic functionCubic(z) whose origin is taken at the position of that adjacent pixel.

That is, when the true green light intensity Lg(I′, J′) is determined bycalculating the weighted sum, the weight is set differently (to a firstor second weight) depending on whether the calculation is performed inthe normal process or the exceptional process.

Although in the eighth embodiment, the cubic function Cubic(z) in theform of equation (39) is used to define the weight depending on thedistance z between a position (I′, J′) of interest and a nearby position(x, y) at which a pixel value Gobs(k, ig, jg) is observed, the weightmay be defined by using another arbitrary function serving as a lowpassfilter in characteristic, such as sin(z)/z.

In the embodiments described above, the first captured image is used asthe reference image, and second to N-th captured images are employed astarget images. Instead of the first captured image, any one of thesecond to N-th captured images may be used as the reference image.

FIG. 54 shows N images obtained by successive N capturing operations (ata high shutter speed). In this specific example shown in FIG. 54, N=8.

First to eighth captured images 401 ₁ to 401 ₈ (k-th image 401 _(k)) arecaptured sequentially in time and displaced in a direction to upperright with passage of time due to camera shake.

FIG. 55 shows eight captured images 401 ₁ to 401 ₈ similar to thoseshown in FIG. 54.

In the case in which the signal processor 7 estimates an output image byusing the first one of the captured images 401 ₁ to 401 ₈ as thereference image and second to eighth images as target images, the outputimage is obtained as an image formed in a coordinate system defined onthe first image 401 ₁ employed as the reference image. In FIG. 55, thefirst image 401 ₁, that is, the output image is represented by a boldline.

The region of the output image represented by the bold line in FIG. 55includes a region 411 located in the upper right corner and denoted by adotted line in which all of eight captured images consisting of first toeighth images 401 ₁ to 401 ₈ can be used in estimation of lightintensity (pixel value) expected to be true. In this region 411, highimage quality can be obtained because estimation of pixel values isperformed using data of all of eight captured images consisting of firstto eighth images 401 ₁ to 401 ₈.

However, in an area of the output image denoted by the bold line in FIG.55 other than the region 411, only data of some of first to eighthcaptured images 401 ₁ to 401 ₈ are available in estimation of theexpected output image. In such an area, image quality in terms ofsharpness becomes low compared with the region 411, because all data ofthe first to eighth captured images 401 ₁ to 401 ₈ cannot be used. Asdescribed above, in a set of captured images having successivedisplacement in the direction to upper right due to camera shake, theamount of data available in estimation of the output image decreaseswith position in an opposite direction, that is, to upper right, and theimage quality in terms of sharpness decreases with respect to thequality of the region 411.

Of data of the second to eighth captured images 401 ₁ to 401 ₈ used astarget images, data in a region 412 outside the output image denoted bythe bold line in FIG. 55 cannot be used in estimation of the outputimage and is discarded uselessly.

As can be understood from the above discussion, when the output image isestimated using the first captured image as the reference image, ifdisplacements occur in a particular direction due to camera shake,degradation in image quality in terms of sharpness occurs in an arealocated away from the center of the output image in a direction oppositeto the direction of displacement, although high image quality isobtained in an area (for example, the region 411 in FIG. 55) locatedaway from the center of the output image in the same direction as thedirection of displacement.

In general, when a user views an image, a central part of the image mostattracts user's attention. Therefore, it is desirable that image qualityin terms of sharpness be high in the central part of the image.

In view of the above, in estimation of an output image, the signalprocessor 7 may employ, as the reference image, an intermediate image ofN successively-captured images. More specifically, of Nsuccessively-captured images, an image captured at the exact or nearlyexact center of the time during which N images were captured is employedas the reference image (hereinafter, such an image will be referred tosimply as an intermediate image), and the other captured images may beused as target images.

For example, as shown in FIG. 56, of eight captured images 401 ₁ to 401₈, the signal processor 7 employs a fourth captured image denoted by abold line as the reference image and estimates an output image. In thiscase, data of all of the eight captured images consisting of first toeighth images 401 ₁ to 401 ₈ are available in a central area 421 whenthe output image is estimated.

As described above, in estimation of an output image, by employing anintermediate image of a plurality of images captured sequentially intime as the reference image and employing the other captured images astarget images, it becomes possible to obtain an output image with higherquality in terms of sharpness in particular in the central area of theoutput image.

In most cases, when a user takes a picture of a subject, the cameraangle is set such that the subject is located at the center of an image(frame). Furthermore, as described above, when a user views an image, acentral part of the image most attracts user's attention. From thispoint of view, an image whose central part is sharper than in peripheralparts can be regarded as a better image than an image that is oppositein terms of sharpness.

In view of the above, as shown in FIG. 56, if an intermediate image isemployed as the reference image and the other images are employed astarget images, data of all of eight captured images consisting of firstto eighth images 401 ₁ to 401 ₈ are used in estimation of a central partof an output image, and the resultant output image has higher quality interms of sharpness in particular in the central part than that obtainedwhen the first captured image is employed as the reference image.

The frequency of camera shake generally falls within the range of 10 to15 Hz. Therefore, in FIG. 56, for a set of eight images 401 ₁ to 401 ₈successively captured at a total effective shutter speed of 1/50 sec orhigher, the displacement due to camera shake can be approximated by alinear displacement. That is, the camera shake can be regarded asmovement in a particular direction at a constant speed. Therefore, forthe set of eight images 401 ₁ to 401 ₈ captured sequentially in time,the displacement due to the camera shake during the exposure time can beapproximated by a linear displacement when the displacement iscompensated for in the estimation of the output image, in which anintermediate image such as a fourth captured image 401 ₄ or a fifthcaptured image 401 ₅ is employed as the reference image therebyobtaining the output image having high image quality in terms ofsharpness in particular in its central part, as described above withreference to FIG. 56.

In the signal processor 7 shown in FIG. 4, when, for example, the fourthcaptured image 401 ₄ of the eight captured images 401 ₁ to 401 ₈ isemployed as the reference image, the fourth captured image 401 ₄employed as the reference image is supplied to the frame memory 22-1 andstored therein, and the first to third captured images 401 ₁ to 401 ₃and the fifth to eighth captured images 401 ₅ to 401 ₈ are respectivelysupplied to the frame memories 22-2 to 22-8 and stored therein.

In addition to high image quality in terms of sharpness in the centralpart of the output image, use of an intermediate image as the referenceimage provides a further advantage that the signal processor 7 can bedesigned more easily.

Herein, let us assume that the displacement due to the camera shake canbe approximated by a linear displacement as described above, and let usalso assume that when eight images 401 ₁ to 401 ₈ are capturedsequentially in time, an image-to-image displacement of, for example,ten pixels occurs. In this case, if the first captured image is employedas the reference image, a displacement of at most 70 pixels can occur,even if the exposure time of each image is infinitely small. Thus, thesignal processor 7 has to be designed so that it can handle adisplacement of at most 70 pixels.

In contrast, when an intermediate image, such as the fourth 401 ₄ imageof the captured images 401 ₁ to 401 ₈ is employed as the referenceimage, the maximum possible displacement is 40 pixels. Therefore, it issufficient for the signal processor 7 to handle a displacement of atmost 40 pixels, and thus it becomes easier to design the signalprocessor 7.

The sequence of processing steps according to the eighth embodiment maybe performed by means of hardware or software, as with the otherembodiments. In the case in which the sequence of processing stepsaccording to the eighth embodiment is performed by means of software,for example, the digital camera 1 may be realized by executing a programon a computer such as that shown in FIG. 41, as with the otherembodiments.

In the present description, the steps described in the program stored inthe storage medium may be performed either in time sequence inaccordance with the order described in the program or in a parallel orseparate fashion.

Although in the embodiments described above, a single-plane sensor isused as the imaging device 4 of the digital camera 1, and an imagehaving three colors for each pixel is produced from an image signalhaving one color signal for each pixel output from the imaging device 4,the imaging device 4 is not limited to the single-plane sensor. Othertypes of imaging device 4 capable of outputting n color signals for eachpixel may be used, and, in the correct image estimation process, animage having (n+1) or more color signals for each pixel may be producedfrom the image signal output from the imaging device 4.

In the embodiments described above, a first captured image or anintermediate captured image is used as the reference image, and apositional relationship between a target image and the reference imageis detected. Alternatively, instead of the first captured image or theintermediate captured image, an arbitrary captured image may be used asthe reference image, and a positional relationship between a targetimage and the reference image may be detected. The present invention maybe applied not only to a digital still camera but also to other types ofcameras such as a digital video camera.

INDUSTRIAL APPLICABILITY

The present invention makes it possible to obtain a sharper image.

1. An image processing method for estimating an output image from aplurality of input images, comprising the steps of: detecting apositional relationship among the plurality of input images captured byimage capture means for capturing an image each pixel of which has npixel values; and estimating the output image by identifying observedpixel components that should be taken into consideration from theplurality of input images for each pixel position of the output image,and producing the output image having (n+1) or more pixel values foreach pixel based on the identified observed pixel components.
 2. Animage processing method according to claim 1, wherein in the step ofestimating the output image, observed pixel components that should betaken into consideration are selected from the plurality of input imagesfor each pixel position of the output image, and the output image having(n+1) or more pixel values for each pixel is produced based on theselected observed pixel components.
 3. An image processing methodaccording to claim 1, wherein in the step of estimating the outputimage, the output image having (n+1) or more color signals for eachpixel is produced from the plurality of input images, based on thepositional relationship detected in the detection step.
 4. An imageprocessing method according to claim 1, wherein in the step ofestimating the output image, the output image is determined taking intoaccount a color correlation at each pixel position and/or in thevicinity of each pixel position of the output image.
 5. An imageprocessing method according to claim 1, wherein in the step ofestimating the output image, pixel values of pixels of the input imagesare changed to pixel values depending on the degree of steepness of anedge, and the output image is estimated based on the resultant changedpixel values.
 6. An image processing method according to claim 1,wherein n is equal to 1; and in the step of estimating the output image,the output image having three pixel values for each pixel is producedfrom the plurality of input images, based on the positional relationshipdetected in the detection step.
 7. An image processing method accordingto claim 1, wherein the image capture means is a single-plane sensor, inwhich a color signal observed varies depending on the position of eachinput image.
 8. An image processing method according to claim 1, whereinthe image capture means captures the plurality of input images withunderexposure; and the image processing method further comprises thestep of correcting the exposure by gaining up each pixel value of theplurality of input images.
 9. An image processing method according toclaim 8, wherein the step of estimating the output image comprises thesteps of: determining a corrected position of each pixel depending onthe positional relationship detected in the detection step for eachpixel of each of the plurality of input images; producing a lightintensity integral constraint formula indicating that the differencebetween the integral of light intensity in a region centered at thecorrected position and a pixel value of a pixel at the correctedposition should be less than a predetermined error; and calculating thelight intensity integral constraint formula.
 10. An image processingmethod according to claim 9, wherein when the light intensity integralconstraint formula is produced, if the plurality of input images havebeen gained up, the error is changed depending on the factor of thegaining up.
 11. An image processing method according to claim 8, whereinin the step of estimating the output image, a corrected position of eachpixel is determined depending on the positional relationship detected inthe detection step for each pixel of each of the plurality of inputimages; and the output image is estimated taking into account a pixelvalue of a pixel of the input images whose corrected position is locatedin the vicinity of each pixel position of the output image and alsotaking into account the error depending on the factor of the gaining up.12. An image processing method according to claim 1, wherein in the stepof estimating the output image, a corrected position of each pixel isdetermined depending on the positional relationship detected in thedetection step for each pixel of each of the plurality of input images;the output image is estimated taking into account a pixel value of apixel of the input images whose corrected position is located in thevicinity of each pixel position of the output image and also taking intoaccount the distance between the corrected position and the position ofthe corresponding pixel of the output image.
 13. An image processingmethod according to claim 1, wherein in the step of estimating theoutput image, a corrected position of each pixel is determined dependingon the positional relationship detected in the detection step for eachpixel of each of the plurality of input images; and the output image isestimated taking into account a pixel value of a pixel of the inputimages whose corrected position is located in the vicinity of each pixelposition of the output image and also taking into account the errordepending on the factor of the gaining up.
 14. An image processingmethod according to claim 1, wherein in the step of estimating theoutput image, a corrected position of each pixel is determined dependingon the positional relationship detected in the detection step for eachpixel of each of the plurality of input images; and the output image isestimated by determining the weighted sum of pixel values of the inputimages whose corrected position is located in the vicinity of each pixelposition of the output image.
 15. An image processing method accordingto claim 14, wherein in the step of estimating the output image, thecorrect pixel value of a pixel of interest is estimated by determiningthe weighted sum of pixel values of the input images whose correctedposition is located in the vicinity of each pixel position of the outputimage, weights of the weighted sum being determined using a functionhaving a lowpass-filter characteristic with respect to the distancebetween the position of the pixel of interest in the output image andthe corrected position.
 16. An image processing method according toclaim 15, wherein in the step of estimating the output image, thecorrect pixel value of the pixel of interest is estimated by selectivelyperforming a first calculation process of determining the weighted sumof pixel values of the input images whose corrected position is locatedin the vicinity of the position of the pixel of interest; or a secondcalculation process of determining the weighted sum of pixel values ofthe input images whose corrected position is located in the vicinity ofthe position of a pixel of the output image located adjacent to thepixel of interest.
 17. An image processing method according to claim 16,wherein in the step of estimating the output image, the secondcalculation process is selected when the sum of weights in the firstcalculation process is equal or nearly equal to
 0. 18. An imageprocessing method according to claim 15, wherein in the step ofestimating the output image, the correct pixel value of the pixel ofinterest is estimated by selectively performing a first calculationprocess of determining the weighted sum of pixel values of the inputimages whose corrected position is located in a first vicinity of theposition of the pixel of interest; or a second calculation process ofdetermining the weighted sum of pixel values of the input images whosecorrected position is located in a second vicinity of the position ofthe pixel of interest, the second vicinity being greater than the firstvicinity.
 19. An image processing method according to claim 18, whereinin the step of estimating the output image, the second calculationprocess is selected when the sum of weights in the first calculationprocess is equal or nearly equal to
 0. 20. An image processing methodaccording to claim 15, wherein in the step of estimating the outputimage, the correct pixel value of the pixel of interest is estimated byselectively performing a first calculation process of determining thesum of pixel values weighted by a first weighting factor of the inputimages whose corrected position is located in the vicinity of theposition of the pixel of interest; or a second calculation process ofdetermining the sum of pixel values weighted by a second weightingfactor of the input images whose corrected position is located in thevicinity of the position of the pixel of interest.
 21. An imageprocessing method according to claim 20, wherein in the step ofestimating the output image, the second calculation process is selectedwhen the sum of weights in the first calculation process is equal ornearly equal to
 0. 22. An image processing method according to claim 1,wherein in the detection step, the positional relationship among theplurality of input images is detected with reference to an input imagecaptured at the exact or nearly exact center of a time during which theplurality of input images were captured.
 23. An image processingapparatus for estimating an output image from a plurality of inputimages, comprising: image capture means for capturing the plurality ofinput images having n pixel values for each pixel; detection means fordetecting a positional relationship among the plurality of input images;and means for estimating the output image by identifying observed pixelcomponents that should be taken into consideration from the plurality ofinput images for each pixel position of the output image, and producingthe output image having (n+1) or more pixel values for each pixel basedon the identified observed pixel components.
 24. A program for causing acomputer to execute a process of estimating an output image from aplurality of input images, the process comprising the steps of:detecting a positional relationship among the plurality of input imagescaptured by image capture means for capturing an image each pixel ofwhich has n pixel values; and estimating the output image by identifyingobserved pixel components that should be taken into consideration fromthe plurality of input images for each pixel position of the outputimage, and producing the output image having (n+1) or more pixel valuesfor each pixel based on the identified observed pixel components.